From ben+python at benfinney.id.au Tue Apr 1 00:01:09 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 01 Apr 2014 09:01:09 +1100 Subject: [Python-ideas] Browser for mailing lists References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> Message-ID: <8561mum3iy.fsf@benfinney.id.au> Nick Coghlan writes: > Mailing lists, I prefer when I'm highly invested. Just as an RSS > reader brings blogs I care about into a single interface, mailing > lists do the same for collaborative communication. Appropriate use of > filters and "mark all as read" then deals with the volume of traffic. > > However, mailing lists are *terrible* if you just want to drop in, ask > one question & leave again. Presumably that's because, in order to follow and participate, one needs to subscribe and receive all the messages in one's email. I agree, that's not ideal for dropping by and asking a question. Which is why I prefer NNTP: it is ideal for *both* these purposes. One can quickly dip into a forum, follow a thread for a while, and unsubscribe just as easily. On the other hand, for high-commitment communication, having all the forums together and threaded is excellent. Having all of this in a single NNTP client interface makes discussion on dozens of forums much more manageable than either ?web forums? (*spit*) or mailing lists. > In the meantime, making GMane more discoverable by suggesting it as an > alternative to subscription in the various list descriptions sounds > reasonable to me. GMane survives on sponsorship and donations. If we're sending more and more people to use that service, perhaps we can suggest supporting them financially too? -- \ ?All opinions are not equal. Some are a very great deal more | `\ robust, sophisticated and well supported in logic and argument | _o__) than others.? ?Douglas Adams | Ben Finney From p.f.moore at gmail.com Tue Apr 1 00:21:26 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 31 Mar 2014 23:21:26 +0100 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: <8561mum3iy.fsf@benfinney.id.au> References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> <8561mum3iy.fsf@benfinney.id.au> Message-ID: On 31 March 2014 23:01, Ben Finney wrote: > Having all of this in a single NNTP client interface makes discussion on > dozens of forums much more manageable than either "web forums" (*spit*) > or mailing lists. One of the key downsides for me of an NNTP interface is that it's (as far as I know) extremely difficult to set up a synchronised environment across two or more PCs. For email, gmail, while not perfect, is a reasonably good client, and I can use it on multiple PCs. I know of no NNTP client with a comparable feature set that tracks what I have and haven't read across multiple PCs, allows posting, etc. If anyone can suggest such a client (which works on Windows, but as it'd probably be web-based, that's probably less of a limitation that normal), I'd be immensely grateful. I realise this is off-topic, but I don't feel *too* guilty as this whole thread is pretty off-topic :-) Paul PS Google groups and the gmane web interface need not apply - neither are particularly friendly (to be polite...) From ncoghlan at gmail.com Tue Apr 1 00:37:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 Apr 2014 08:37:52 +1000 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> <8561mum3iy.fsf@benfinney.id.au> Message-ID: On 1 Apr 2014 08:22, "Paul Moore" wrote: > > On 31 March 2014 23:01, Ben Finney wrote: > > Having all of this in a single NNTP client interface makes discussion on > > dozens of forums much more manageable than either "web forums" (*spit*) > > or mailing lists. > > One of the key downsides for me of an NNTP interface is that it's (as > far as I know) extremely difficult to set up a synchronised > environment across two or more PCs. Yep, handling multiple systems is when I stopped using it, too. Phones make it worse (a lot of my email list time is on the bus these days - aside from making bulk deletion while still retaining some context painful, Gmail handles that pretty well) > For email, gmail, while not > perfect, is a reasonably good client, and I can use it on multiple > PCs. I know of no NNTP client with a comparable feature set that > tracks what I have and haven't read across multiple PCs, allows > posting, etc. If anyone can suggest such a client (which works on > Windows, but as it'd probably be web-based, that's probably less of a > limitation that normal), I'd be immensely grateful. I realise this is > off-topic, but I don't feel *too* guilty as this whole thread is > pretty off-topic :-) Improving the accessibility of the list is pretty close to on topic when it's covering some specific ideas that could improve things without much work. I only see it as a problem when we head off on wild impractical tangents that are unlikely to happen any time soon (if ever) due to resource constraints :) Cheers, Nick. > > Paul > > PS Google groups and the gmane web interface need not apply - neither > are particularly friendly (to be polite...) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Tue Apr 1 00:44:32 2014 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 1 Apr 2014 09:44:32 +1100 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: <20140331134858.4781ff08@anarchist.wooz.org> References: <20140331134858.4781ff08@anarchist.wooz.org> Message-ID: <20140331224432.GA19883@cskk.homeip.net> On 31Mar2014 13:48, Barry Warsaw wrote: > On Mar 31, 2014, at 04:52 PM, Cameron Simpson wrote: > >I'm wedded to having the archives available in a simple and easy > >to fetch form, in a format that is easily imported by many MUAs. > >Therefore: mbox. > > I'm not sure that's really an effective use case. It is for me. I do this regularly. I've come across this in the past: one describes one's work practices and people tell you it isn't a real world use case. I know I don't _want_ to live in the real world, but I'm pretty sure I do. > For example, python-list > raw mbox archives are 156GB. I'm positive it's highly compressable, but > still, why would you want to download some huge mbox like that? As I've said repeatedly now. To be able to search it or otherwise consult it with my own tools and when offline, and to browse it the way I _like_ to browse email: in a mail reader. BTW: 156 GB? Only python-list? I'd have thought I'd have pulled it when I joined and my archive is only around 2GB as stored on disc. A rough guess based on eyeballing the gzipped mbox downloads available from pipermail suggests a similar total. Are there tons of predating 1999, or not in pipermail? Cheers, -- Cameron Simpson One rider (Kenny Roberts?) described The CorkScrew as: It's like driving your bike into a phone booth, and then having the phone booth dropped down an elevator shaft! From rosuav at gmail.com Tue Apr 1 01:22:48 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Apr 2014 10:22:48 +1100 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> Message-ID: On Tue, Apr 1, 2014 at 8:51 AM, Nick Coghlan wrote: > However, mailing lists are *terrible* if you just want to drop in, ask one > question & leave again. One possible answer to that is AskBot (a self-hosted > site-specific, Stack Overflow style Q&A forum), but the MM3/HyperKitty web > gateway idea is aimed at making the existing *already invested* list > participants more readily available to new users without having to ask those > existing participants to change their reading habits and look in a new place > for questions. Drifting off-topic, but I think this is actually a subtle feature. I've often joined a mailing list to ask one specific question, and then seen questions that I can help answer. In fact, that's how I came to be stuck on python-list and family... of course, other people may not see that as a positive feature... What it means is that there's no stark difference between "people here to ask questions" and "people here to answer and help out". It's users helping users. Most lists don't have so much traffic that it turns me off (a few do - I unsubscribed from the SciTE lists fairly quickly), so I'll hang around for a while after getting my answer - or, given that I've probably exhausted a number of other avenues before even posting the question, after not getting any answer. The more people you can engage that way, the more likely the list will have the person on it who can answer that obscure question that comes up. ChrisA From steve at pearwood.info Tue Apr 1 02:21:07 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 1 Apr 2014 11:21:07 +1100 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> Message-ID: <20140401002107.GQ16526@ando> On Mon, Mar 31, 2014 at 01:09:48PM +0100, Richard Prosser wrote: > I agree - links to search tools or at least explanations of how to use them > would be very welcome. It's 2014, not 1998. Do we really need to teach people how to use the Internet? Not just people in general, for that matter, but *programmers*? -- Steven From ethan at stoneleaf.us Tue Apr 1 02:31:44 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 31 Mar 2014 17:31:44 -0700 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: <20140401002107.GQ16526@ando> References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <20140401002107.GQ16526@ando> Message-ID: <533A08F0.9020305@stoneleaf.us> On 03/31/2014 05:21 PM, Steven D'Aprano wrote: > On Mon, Mar 31, 2014 at 01:09:48PM +0100, Richard Prosser wrote: >> >> I agree - links to search tools or at least explanations of how to use them >> would be very welcome. > > It's 2014, not 1998. Do we really need to teach people how to use the > Internet? Not just people in general, for that matter, but > *programmers*? Yes. -- ~Ethan~ From ncoghlan at gmail.com Tue Apr 1 07:50:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 Apr 2014 15:50:21 +1000 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: <533A08F0.9020305@stoneleaf.us> References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <20140401002107.GQ16526@ando> <533A08F0.9020305@stoneleaf.us> Message-ID: On 1 April 2014 10:31, Ethan Furman wrote: > On 03/31/2014 05:21 PM, Steven D'Aprano wrote: >> On Mon, Mar 31, 2014 at 01:09:48PM +0100, Richard Prosser wrote: >>> I agree - links to search tools or at least explanations of how to use >>> them >>> would be very welcome. >> >> It's 2014, not 1998. Do we really need to teach people how to use the >> Internet? Not just people in general, for that matter, but >> *programmers*? > > Yes. The other thing to keep in mind is that we're actively trying to remove barriers to entry to participating in the core development process, while still respecting the perspectives of long term community members. That's a long, slow, process with very few quick fixes available, but providing pointers to useful tools (like gmane) to help make them more discoverable is certainly something we can do in the meantime. (We do already reference it from https://docs.python.org/devguide/communication.html#mailing-lists, but that page has its own discoverability issues) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Tue Apr 1 13:58:43 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 1 Apr 2014 22:58:43 +1100 Subject: [Python-ideas] Default value for input Message-ID: <20140401115842.GR16526@ando> (I'm talking about Python 3 input, previously known as raw_input.) I often have the need to include a default or initial value when asking the user for input. In other words, I'd like to call this: input("What is the fish of the day? ", "trout a la creme") and have the prompt be "What is the fish of the day? " and the initial value in the edit buffer be "trout a la creme", fully editable with whatever line editing tools are available on the user's system. There are work-arounds for this lack, but they don't make a nice clean UI. Under Linux, with readline, I can implement this relatively simply: _input = input def input(prompt, initial=''): readline.set_startup_hook(lambda: readline.insert_text(initial)) try: return _input(prompt) finally: readline.set_startup_hook(None) Aside: I'm a bit concerned about unconditionally setting the startup hook to None, rather than restoring whatever it was before I messed with it. But I see no way to save the startup hood from Python code. However this doesn't work under Windows, or any other system where readline is not available. Does anyone else think this would be a useful feature? I see a few questions about this in various places, such as StackOverflow: http://stackoverflow.com/questions/5403138/how-to-set-a-default-string-for-raw-input but I don't see that it has ever been requested on the bug tracker. Is it practical to include an editable initial value on systems without readline? If a platform-independent version is not practical, I don't think there's any point in only half-supporting it. -- Steven From rosuav at gmail.com Tue Apr 1 14:11:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 1 Apr 2014 23:11:47 +1100 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401115842.GR16526@ando> References: <20140401115842.GR16526@ando> Message-ID: On Tue, Apr 1, 2014 at 10:58 PM, Steven D'Aprano wrote: > I often have the need to include a default or initial value when > asking the user for input. In other words, I'd like to call this: > > input("What is the fish of the day? ", "trout a la creme") > > > and have the prompt be "What is the fish of the day? " and the initial > value in the edit buffer be "trout a la creme", fully editable with > whatever line editing tools are available on the user's system. There > are work-arounds for this lack, but they don't make a nice clean UI. > > Is it practical to include an editable initial value on systems without > readline? If a platform-independent version is not practical, I don't > think there's any point in only half-supporting it. You could fudge it on a non-readline system by rewriting the prompt and turning blank into the default: _input = input def input(prompt, initial=''): if not initial: return _input(prompt) return _input("%s[%s] "%(prompt, initial)) or initial That's not the same as an *editable* initial value, but it's a visible default, which is a useful feature. However, the convention is usually to put the square brackets before the punctuation: What is the fish of the day [trout a la creme]? so it might be better to make something more sophisticated that recognizes question marks and colons and shifts the default marker before them. Sounds like a reasonable feature, to me. I've no idea about libedit, which is the most common non-readline system that has real editing; you might need to have several options and then a final fall-back that looks like the above, but it should be possible. How common is the square-brackets-means-default convention? Would, for instance, a Windows-only user understand it? ChrisA From steve at pearwood.info Tue Apr 1 15:01:47 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 Apr 2014 00:01:47 +1100 Subject: [Python-ideas] Default value for input In-Reply-To: References: <20140401115842.GR16526@ando> Message-ID: <20140401130147.GT16526@ando> On Tue, Apr 01, 2014 at 11:11:47PM +1100, Chris Angelico wrote: > On Tue, Apr 1, 2014 at 10:58 PM, Steven D'Aprano wrote: > > I often have the need to include a default or initial value when > > asking the user for input. In other words, I'd like to call this: > > > > input("What is the fish of the day? ", "trout a la creme") > > > > > > and have the prompt be "What is the fish of the day? " and the initial > > value in the edit buffer be "trout a la creme", fully editable with > > whatever line editing tools are available on the user's system. There > > are work-arounds for this lack, but they don't make a nice clean UI. ....^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :-) > You could fudge it on a non-readline system by rewriting the prompt > and turning blank into the default: Yes, I have work-arounds. They aren't what you call elegant. Your suggestion, to wrap the default in square brackets: What is the fish of the day? [trout a la creme] requires the user to know that pressing Enter will return "trout a la creme", rather than it meaning that "trout a la creme" is optional, which is another common meaning for square brackets. That's actually what I do, except I prefer < > angle brackets. It also doesn't help them if they want to enter "trout a la creme with salad". Any change at all, even a single character, means that they have to type the entire string from scratch. Not user-friendly at all. Anyway, I'm not looking for work-arounds. I know how to program work-arounds :-) I'm asking if this is likely to be both practical and useful enough to be worth making an enhancement request. > Sounds like a reasonable feature, to me. I've no idea about libedit, > which is the most common non-readline system that has real editing; > you might need to have several options and then a final fall-back that > looks like the above, but it should be possible. libedit is, at least theoretically, identical in functionality to readline, only with different syntax. If the platform that Python is running on has no readline/libedit (which means Windows), or if the person building Python from source disables readline/libedit, the only editing tool which is available is Good Old Backspace. But that's better than nothing. I'm not asking for input() to duplicate all of readline, just to start with an optional initial value in the edit buffer. -- Steven From rosuav at gmail.com Tue Apr 1 15:19:24 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 2 Apr 2014 00:19:24 +1100 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401130147.GT16526@ando> References: <20140401115842.GR16526@ando> <20140401130147.GT16526@ando> Message-ID: On Wed, Apr 2, 2014 at 12:01 AM, Steven D'Aprano wrote: > On Tue, Apr 01, 2014 at 11:11:47PM +1100, Chris Angelico wrote: >> On Tue, Apr 1, 2014 at 10:58 PM, Steven D'Aprano wrote: >> > I often have the need to include a default or initial value when >> > asking the user for input. In other words, I'd like to call this: >> > >> > input("What is the fish of the day? ", "trout a la creme") >> > >> > >> > and have the prompt be "What is the fish of the day? " and the initial >> > value in the edit buffer be "trout a la creme", fully editable with >> > whatever line editing tools are available on the user's system. There >> > are work-arounds for this lack, but they don't make a nice clean UI. > ....^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > :-) > > Anyway, I'm not looking for work-arounds. I know how to program > work-arounds :-) I'm asking if this is likely to be both practical and > useful enough to be worth making an enhancement request. Well, your subject line does say "default", which is a concept that can be rendered. "Editable default" is a much bigger feature, and requires actual editing keys. It would make a small amount of sense to have redirected input able to use defaults by having blank lines in the input. >> Sounds like a reasonable feature, to me. I've no idea about libedit, >> which is the most common non-readline system that has real editing; >> you might need to have several options and then a final fall-back that >> looks like the above, but it should be possible. > > libedit is, at least theoretically, identical in functionality to > readline, only with different syntax. > > If the platform that Python is running on has no readline/libedit (which > means Windows), or if the person building Python from source disables > readline/libedit, the only editing tool which is available is Good Old > Backspace. But that's better than nothing. I'm not asking for input() to > duplicate all of readline, just to start with an optional initial value > in the edit buffer. So what I'd see, then, is that your function would need three, maybe four, options: 1) Do we have readline? If so, do what you said up top. 2) Do we have libedit? If so, do the same thing with different syntax. 3) Can we pre-fill the input field? If so, do. 4) Put the default in brackets (square or angle) and translate blank into that. (Maybe say "Prompt? [blank=default] " for a bit more clarity.) If the third option is guaranteed as part of the implementation, then (obviously) the fourth isn't needed. This sounds like an excellent little feature. Burying that complexity away inside input() would help enormously with cross-platform usability; having the third and maybe fourth options there would mean that a programmer who uses readline on Linux can still be confident that his program will work, if a bit more clunkily, on Windows. I'm not offering this as a work-around but as a fall-back; answering your initial concern about platform-independence. Python is very good at hiding platform-specific details (import subprocess and don't concern yourself with CreateProcess() vs fork()/exec(), etc etc); abstracting away the difference between a really nice readline implementation and a messy-but-workable pure prompt implementation gives your program graceful degradation with a very simple notation (just add another arg to input()). ChrisA From ncoghlan at gmail.com Tue Apr 1 15:26:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 Apr 2014 23:26:50 +1000 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> Message-ID: On 31 March 2014 02:05, Guido van Rossum wrote: > On Sat, Mar 29, 2014 at 7:17 PM, Nick Coghlan wrote: >> This PEP proposes a number of small adjustments to the APIs of the >> ``bytes`` >> and ``bytearray`` types to make their behaviour more internally consistent >> and to make it easier to operate entirely in the binary domain > > > I hope you don't mind I cut the last 60% of this sentence (everything after > "binary domain"). No worries - the shorter version is better. I've been spending too much time in recent months explaining the significant of the text model changes to different people, so now I end up trying to explain it any time I write about it :) >> Background >> ========== >> >> Over the course of Python 3's evolution, a number of adjustments have been >> made to the core ``bytes`` and ``bytearray`` types as additional practical >> experience was gained with using them in code beyond the Python 3 standard >> library and test suite. However, to date, these changes have been made >> on a relatively ad hoc tactical basis as specific issues were identified, >> rather than as part of a systematic review of the APIs of these types. > > > I'm not sure you can claim that. We probably have more information based on > experience now than when we did the redesign. (At that time most experience > was based on using str() for binary data.) Yeah, I was mostly thinking of the change to make the search APIs accept both integers and subsequences when I wrote that. I'll try to think of a better way of wording it. I also realised in reviewing the docs that a key part of the problem may actually be a shortcut I took in the sequence docs rewrite that I did quite a while ago now - the bytes/bytearray docs are currently written in terms of *how they differ from str*. They don't really cover the "container of integers" aspect particularly well. I now believe a better way to tackle that would to be upfront that these types basically have two APIs on a single class: their core tuple/list of integers "arbitrary binary data" API, and then the str-inspired "binary data with ASCII segments" API on top of that. >> This >> approach has allowed inconsistencies to creep into the API design as to >> which >> input types are accepted by different methods. Additional inconsistencies >> linger from an earlier pre-release design where there was *no* separate >> ``bytearray`` type, and instead the core ``bytes`` type was mutable (with >> no immutable counterpart), as well as from the origins of these types in >> the text-like behaviour of the Python 2 ``str`` type. > > You make it sound as if modeling bytes() after Python 2's str() was an > accident. It wasn't. Sorry, didn't mean to imply that. More that we hadn't previously sat down and thought through how best to clearly articulate this in the docs, and hence some of the current inconsistencies hadn't become clear. >> This PEP aims to provide the missing systematic review, with the goal of >> ensuring that wherever feasible (given backwards compatibility >> constraints) >> these current inconsistencies are addressed for the Python 3.5 release. > > I would like to convince you to aim lower, drop the "systematic review", and > just focus on some changes that are likely to improve users' experience > (which includes porting Python 2 code). After re-reading the current docs (which I also wrote, but this aspect was a mere afterthought at the time), I'm thinking a useful course of action in parallel with this PEP will be for me to work on improving the Python 3.4 docs for these types. The bits I find too hard to explain then become fodder for tweaks in 3.5. >> Proposals >> ========= >> * more consistently accepting length 1 ``bytes`` objects as input where an >> integer between ``0`` and ``255`` inclusive is expected, and vice-versa > > > Not sure I like this as a goal. OK, stronger: I don't like this goal. Roger. As noted above, I now think we can address this by splitting the API documentation instead, so that there's a clear "tuple/list of ints" section and a "binary data with ASCII segments" section. Some hybrid APIs (like the search ones) may appear in both. In terms of analogies to other types: Behaviour is common to tuple + str: hybrid API for bytes + bytearray Behaviour is list-only: int-only API for bytearray Behaviour is str-only: str-like only API for bytes + bytearray Now that I've framed the question that way, I think I can not only make it make sense in the docs, but I believe the 3.4 behaviour is already pretty close to consistent with it. The proposed bytes.byte() constructor would then more cleanly handle the few cases where it may be desirable to pass an int to a str-like API (such as replace()) >> * allowing users to easily convert integer output to a length 1 ``bytes`` >> object > > I think you meant integer values instead of output? Sort of - I was thinking of reversing the effects of indexing here. That is, replacing the current: x = data[0:1] with: x = bytes.byte(data[0]) > In Python 2 we did this > with the global function chr(), but in Python 3 that creates a str(). (The > history of chr() and ord() sa built-in functions is that they long predates > the notion of methods (class- or otherwise), and their naming comes straight > from Pascal.) > > Anyway, I don't know that the use case is so common that it needs more than > bytes([i]) or bytearray([i]) -- if there is an argument to be made for > bytes.byte(i) and bytearray.byte(i) it would be that the [i] in the > constructor is somewhat hard to grasp. Since I was mostly thinking about an alternative to slicing to convert an index lookup back to a bytes object, this doesn't seem appealing to me: x = bytes([data[0]]) The other one is that "bytes([i])" doesn't play nice with higher order functions like map. I don't expect wanting this to be *hugely* common, but I do think there's value in having the primitive conversion operation implied by the constructor behaviour available as a Python level operation. >> Alternate Constructors >> ---------------------- >> >> The ``bytes`` and ``bytearray`` constructors currently accept an integer >> argument, but interpret it to mean a zero-filled object of the given >> length. > > > This is one of the two legacies of the original "mutable bytes" design, and > I agree we should strive to replace it -- although I think one round of > deprecation may be too quick. Postponing removal to 3.7 or indefinitely is fine by me. While I think it should go away, I'm in no hurry to get rid of it - it started bothering me less once I realised you can already safely call bytes on arbitrary objects by passing them through memoryview first (as that doesn't have the mutable legacy that causes problems with integer input). >> For ``bytes``, a ``byte`` constructor is proposed that converts integers >> (as indicated by ``operator.index``) > > I know why you reference this, but it feels confusing to me. At this point > in the narrative it's better to just say "integer" and explain how it > decides "integer-ness" later. Sounds good (I actually meant to double check that we *do* currently accept arbitrary integer-like objects in the bytes constructor). >> in the appropriate range to a ``bytes`` >> object, converts objects that support the buffer API to bytes, and also >> passes through length 1 byte strings unchanged:: > > I think the second half (accepting bytes instances of length 1) is wrong > here and doesn't actually have a practical use case. I'll say more below. Agreed, I now think "the binary equivalent of chr()" would be much better behaviour here. >> For ``bytearray``, a ``from_len`` constructor is proposed that >> preallocates >> the buffer filled with a particular value (default to ``0``) as a direct >> replacement for the current constructor behaviour, rather than having to >> use >> sequence repetition to achieve the same effect in a less intuitive way:: >> >> >>> bytearray.from_len(3) >> bytearray(b'\x00\x00\x00') >> >>> bytearray.from_len(3, 6) >> bytearray(b'\x06\x06\x06') >> >> This part of the proposal was covered by an existing issue >> [empty-buffer-issue]_ and a variety of names have been proposed >> (``empty_buffer``, ``zeros``, ``zeroes``, ``allnull``, ``fill``). The >> specific name currently proposed was chosen by analogy with >> ``dict.fromkeys()`` and ``itertools.chain.from_iter()`` to be completely >> explicit that it is an alternate constructor rather than an in-place >> mutation, as well as how it differs from the standard constructor. > > I think you need to brainstorm more on the name; from_len() looks pretty > awkward. And I think it's better to add it to bytes() as well, since the two > classes intentionally try to be as similar as possible. I initially liked Barry's "fill" suggestion, but then realised it read too much like an in-place operation (at least to my mind). Here are some examples (using Brett's suggestion of a keyword only second parameter): bytearray.zeros(3) # NumPy spelling, no configurable fill value bytearray.fill(3) bytearray.fill(3, fillvalue=6) bytearray.filled(3) bytearray.filled(3, fillvalue=6) To be honest, I'm actually coming around to the "just copy the 'zeros' name from NumPy and call it done" view on this one. I don't have a concrete use case for a custom fill value, and I think I'll learn quickly enough that it uses the shorter spelling. >> Open questions >> ^^^^^^^^^^^^^^ >> * Should ``bytes.byte()`` raise ``TypeError`` or ``ValueError`` for binary >> sequences with more than one element? The ``TypeError`` currently >> proposed >> is copied (with slightly improved wording) from the behaviour of >> ``ord()`` >> with sequences containing more than one code point, while ``ValueError`` >> would be more consistent with the existing handling of out-of-range >> integer values. > > It should not accept any bytes arguments. But if somehow you convince me > otherwise, it should be ValueError (and honestly, ord() is wrong there). > >> >> * ``bytes.byte()`` is defined above as accepting length 1 binary sequences >> as individual bytes, but this is currently inconsistent with the main >> ``bytes`` constructor:: >> >> >>> bytes([b"a", b"b", b"c"]) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: 'bytes' object cannot be interpreted as an integer >> >> Should the ``bytes`` constructor be changed to accept iterables of >> length 1 >> bytes objects in addition to iterables of integers? If so, should it >> allow a mixture of the two in a single iterable? > > Noooooooooooooooooooooooooo!!!!! Yeah, it bothered me, too :) As you suggest, I think it makes sense to extrapolate this the other way and change the definition of bytes.byte() to be a true inverse of ord() for binary data. >> Iteration >> --------- >> >> Iteration over ``bytes`` objects and other binary sequences produces >> integers. Rather than proposing a new method that would need to be added >> not only to ``bytes``, ``bytearray`` and ``memoryview``, but potentially >> to third party types as well, this PEP proposes that iteration to produce >> length 1 ``bytes`` objects instead be handled by combining ``map`` with >> the new ``bytes.byte()`` alternate constructor proposed above:: >> >> for x in map(bytes.byte, data): >> # x is a length 1 ``bytes`` object, rather than an integer >> # This works with *any* container of integers in the range >> # 0 to 255 inclusive > > I can see why you don't like a new method, but this idiom is way too verbose > and unintuitive to ever gain traction. Let's just add a new method to all > three types, 3rd party types will get the message. Fair enough. Is "iterbytes()" OK as the name?: for x in date.iterbytes(): # x is a length 1 ``bytes`` object, rather than an integer # This works with *any* container of integers in the range # 0 to 255 inclusive >> Consistent support for different input types >> -------------------------------------------- >> >> In Python 3.3, the binary search operations (``in``, ``count()``, >> ``find()``, ``index()``, ``rfind()`` and ``rindex()``) were updated to >> accept integers in the range 0 to 255 (inclusive) as their first argument >> (in addition to the existing support for binary sequences). > > > I wonder if that wasn't a bit over-zealous. While 'in', count() and index() > are sequence methods (looking for elements) that have an extended meaning > (looking for substrings) for string types, the find() and r*() variants are > only defined for strings. I suspect they're using the same underlying search code, although I haven't actually checked. >> This PEP proposes extending that behaviour of accepting integers as being >> equivalent to the corresponding length 1 binary sequence to several other >> ``bytes`` and ``bytearray`` methods that currently expect a ``bytes`` >> object for certain parameters. In essence, if a value is an acceptable >> input to the new ``bytes.byte`` constructor defined above, then it would >> be acceptable in the roles defined here (in addition to any other already >> supported inputs): >> >> * ``startswith()`` prefix(es) >> * ``endswith()`` suffix(es) >> >> * ``center()`` fill character >> * ``ljust()`` fill character >> * ``rjust()`` fill character >> >> * ``strip()`` character to strip >> * ``lstrip()`` character to strip >> * ``rstrip()`` character to strip >> >> * ``partition()`` separator argument >> * ``rpartition()`` separator argument >> >> * ``split()`` separator argument >> * ``rsplit()`` separator argument >> >> * ``replace()`` old value and new value >> >> In addition to the consistency motive, this approach also makes it easier >> to work with the indexing behaviour , as the result of an indexing >> operation >> can more easily be fed back in to other methods. > > > I think herein lies madness. The intention seems to be to paper over as much > as possible the unfortunate behavior of b[i]. But how often does any of > these methods get called with such a construct? And how often will that be > in a context where this is the *only* thing that is affected by b[i] > returning an int in Python 3 but a string in Python 2? (In my experience > these are mostly called with literal arguments, except inside wrapper > functions that are themselves intended to be called with a literal > argument.) Weakening the type checking here seems a bad idea -- it would > accept integers in *any* context, and that would just cause more nasty > debugging issues. Yeah, I agree with this now. I was already starting to get "What's the actual use case here?" vibes while writing the PEP, and your reaction makes it easy to change my mind :) "replace()" seems like the only one where a reasonable case might be made to allowing integer input (and that's actually the one Brandon was asking about that got me thinking along these lines in the first place). >> For ``bytearray``, some additional changes are proposed to the current >> integer based operations to ensure they remain consistent with the >> proposed >> constructor changes:: >> >> * ``append()``: updated to be consistent with ``bytes.byte()`` >> * ``remove()``: updated to be consistent with ``bytes.byte()`` >> * ``+=``: updated to be consistent with ``bytes()`` changes (if any) >> * ``extend()``: updated to be consistent with ``bytes()`` changes (if any) > > > Eew again. These are operations from the MutableSequence ABC and there is no > reason to make their signatures fuzzier. >> Acknowledgement of surprising behaviour of some ``bytearray`` methods >> --------------------------------------------------------------------- >> >> Several of the ``bytes`` and ``bytearray`` methods have their origins in >> the >> Python 2 ``str`` API. > > You make it sound as if this is a bad thing or an accident. Again, not my intention. I think that impression will be easier to avoid once the PEP is recast as treating the issue as primarily a documentation problem, with just a few minor API tweaks. >> As ``str`` is an immutable type, all of these >> operations are defined as returning a *new* instance, rather than >> operating >> in place. This contrasts with methods on other mutable types like >> ``list``, >> where ``list.sort()`` and ``list.reverse()`` operate in-place and return >> ``None``, rather than creating a new object. > > So does bytestring.reverse(). And if you really insist we can add > bytestring.sort(). :-) Yeah, this all becomes substantially *less* surprising once these types are documented as effectively exposing two mostly distinct APIs (their underlying "container of ints" API for arbitrary binary data, and then the additional str-like API for binary data with ASCII compatible segments) I'm not sure when I'll get the PEP updated (since this isn't an urgent problem, I just wanted to get the initial draft of the PEP written while the problem was fresh in my mind), but I think the end result should be relatively non-controversial once I incorporate your feedback. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From random832 at fastmail.us Tue Apr 1 15:33:38 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 01 Apr 2014 09:33:38 -0400 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401115842.GR16526@ando> References: <20140401115842.GR16526@ando> Message-ID: <1396359218.17911.101416629.44EF5357@webmail.messagingengine.com> On Tue, Apr 1, 2014, at 7:58, Steven D'Aprano wrote: > However this doesn't work under Windows, or any other system where > readline is not available. It is possible, though awkward, to pre-populate the input buffer on Windows using WriteConsoleInput. There's probably a million other things that are higher priority to do with the windows console before this, though (starting with unicode support). From random832 at fastmail.us Tue Apr 1 15:41:40 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 01 Apr 2014 09:41:40 -0400 Subject: [Python-ideas] Subequence search (was: Fixing the Python 3 bytes constructor) In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> Message-ID: <1396359700.20750.101419865.55C1D916@webmail.messagingengine.com> On Tue, Apr 1, 2014, at 9:26, Nick Coghlan wrote: > Yeah, I was mostly thinking of the change to make the search APIs > accept both integers and subsequences when I wrote that. I'll try to > think of a better way of wording it. I have a mostly unrelated question. Is there a _general_ subsequence search function [that works on lists or even iterables], or is it only available for str and bytes? From p.f.moore at gmail.com Tue Apr 1 16:21:40 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 1 Apr 2014 15:21:40 +0100 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401130147.GT16526@ando> References: <20140401115842.GR16526@ando> <20140401130147.GT16526@ando> Message-ID: On 1 April 2014 14:01, Steven D'Aprano wrote: > If the platform that Python is running on has no readline/libedit (which > means Windows), or if the person building Python from source disables > readline/libedit, the only editing tool which is available is Good Old > Backspace. But that's better than nothing. I'm not asking for input() to > duplicate all of readline, just to start with an optional initial value > in the edit buffer. Windows has much richer line editing capabilities than "plain old backspace" - sufficiently so that I end up very frustrated when working on Unix programs that don't have readline support, because they are *so* limited (no arrow keys? seriously?). Ignoring that point, though, pre-filled edit buffers are *extremely* uncommon on Windows, and I would imagine them being very surprising to a user. I'm not sure I've ever seen them on Unix either, though, so maybe the target audience for this has specialised or unusual expectations? Personally, I'm used to (and comfortable with) the "Enter a value [12]: " convention where this means that pressing enter without providing a value gives 12 (and I'd just implement that manually with if val='': val = '12'). I'm not sure how easy it would be to implement for Windows. I've never needed it, so I've never looked into it. -1 on it being a builtin capability of input(). -0 on it being a separate builtin or stdlib function (mainly because I don't think it's needed often enough to be worth it). It seems to me it'd be fine as a package on PyPI, or maybe a recipe. Paul. From techtonik at gmail.com Tue Apr 1 15:20:10 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 1 Apr 2014 16:20:10 +0300 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: References: Message-ID: On Sat, Mar 29, 2014 at 6:29 PM, Richard Prosser wrote: > I wanted to search this list (and other Python ones) but it is practically > impossible to do so. Welcome to the museum. Please turn off your gadgets and mobile phones during your visit. =) Seriously, I use this for searching Python-ideas: https://groups.google.com/forum/#!forum/python-ideas > So why not add a tool like http://markmail.org/search/?q=python to > https://mail.python.org/mailman/listinfo/python-ideas or the like? I got > this from http://openmeetings.apache.org/mail-lists.html 4 years ago I brought this question on python-dev: http://www.gossamer-threads.com/lists/python/dev/807336?do=post_view_threaded No generic search form was added to mailman lists, but at least a link to GMane appeared. I'd say that what you see is a way how things are (I am not permitted to say "how bad things" are, but if you're an UX expert, just to let you know my opinion). Nick said: > we're actively trying to > remove barriers to entry to participating in the core development > process, while still respecting the perspectives of long term > community members" I believe that this perfectly sums up what's going on. I am not a community member (didn't sign up CoC or CLA), but I already have a problem trying to keep up with all the threads and mails around Python. Limiting interfaces for collaboration it a natural process to prevent traffic increase. In terms of systems theory - resistance to change old interfaces helps save energy, so there is potentially more resistance than is visible. From techtonik at gmail.com Tue Apr 1 15:31:15 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 1 Apr 2014 16:31:15 +0300 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: <20140401002107.GQ16526@ando> References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <20140401002107.GQ16526@ando> Message-ID: On Tue, Apr 1, 2014 at 3:21 AM, Steven D'Aprano wrote: > On Mon, Mar 31, 2014 at 01:09:48PM +0100, Richard Prosser wrote: >> I agree - links to search tools or at least explanations of how to use them >> would be very welcome. > > It's 2014, not 1998. Do we really need to teach people how to use the > Internet? Not just people in general, for that matter, but > *programmers*? At least there is a problem to teach people born in 1998 how to use means of communications that predates mobile phone era. I'd say that old people like me should not underestimate how new generation-m is different. It may be that ML is the only way to effectively communicate but that doesn't mean they can use it. From techtonik at gmail.com Tue Apr 1 15:49:34 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 1 Apr 2014 16:49:34 +0300 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: <8561mum3iy.fsf@benfinney.id.au> References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> <8561mum3iy.fsf@benfinney.id.au> Message-ID: On Tue, Apr 1, 2014 at 1:01 AM, Ben Finney wrote: > > GMane survives on sponsorship and donations. If we're sending more and > more people to use that service, perhaps we can suggest supporting them > financially too? GMane is a good project, but is it open source? Does it help people to raise expertise in Python, train they skills and in the end contribute to engineer the next generation scripting language? I thought that the goal of Python folks as a community is to encourage collaboration, reuse and exchange of experience. In the regard I expected people to propose integrating something like https://bitbucket.org/mchaput/whoosh/wiki/Home and discussing what is required to make it possible. -- anatoly t. From breamoreboy at yahoo.co.uk Tue Apr 1 17:30:34 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 01 Apr 2014 16:30:34 +0100 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401115842.GR16526@ando> References: <20140401115842.GR16526@ando> Message-ID: On 01/04/2014 12:58, Steven D'Aprano wrote: > > However this doesn't work under Windows, or any other system where > readline is not available. > Is pyreadline an option for you? Anything you could borrow from iPython or similar? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From barry at python.org Tue Apr 1 17:30:36 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 1 Apr 2014 11:30:36 -0400 Subject: [Python-ideas] Browser for mailing lists References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> <8561mum3iy.fsf@benfinney.id.au> Message-ID: <20140401113036.421fafd2@anarchist.wooz.org> On Apr 01, 2014, at 09:01 AM, Ben Finney wrote: >Which is why I prefer NNTP: it is ideal for *both* these purposes. One >can quickly dip into a forum, follow a thread for a while, and >unsubscribe just as easily. On the other hand, for high-commitment >communication, having all the forums together and threaded is excellent. Yes, exactly. >Having all of this in a single NNTP client interface makes discussion on >dozens of forums much more manageable than either ?web forums? (*spit*) >or mailing lists. My only beef with Gmane is that it should be able to maintain a global reputation service. E.g. if I post to a few newsgroups with my python.org address, then after a N-confirmation dances, it should allow me to post to any newsgroup (well, any that is linked to an upstream list with a compatible posting policy). That's one of the things that an NNTP interface to MM3 would allow, and I think it would work more smoothly. Of course, the nice thing about Gmane is that it collates lists across many administrative domains, so I don't think an NNTP-aware MM3 would replace Gmane. (The other thing I want to do is make it one-click for a list admin to opt into Gmane. That actually should be pretty easy, and maybe the default for public discussion lists.) >> In the meantime, making GMane more discoverable by suggesting it as an >> alternative to subscription in the various list descriptions sounds >> reasonable to me. > >GMane survives on sponsorship and donations. If we're sending more and >more people to use that service, perhaps we can suggest supporting them >financially too? +1. They're good folks and provide the entire FLOSS world a huge benefit. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From barry at python.org Tue Apr 1 17:32:39 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 1 Apr 2014 11:32:39 -0400 Subject: [Python-ideas] Browser for mailing lists References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> <8561mum3iy.fsf@benfinney.id.au> Message-ID: <20140401113239.57ae0b62@anarchist.wooz.org> On Mar 31, 2014, at 11:21 PM, Paul Moore wrote: >One of the key downsides for me of an NNTP interface is that it's (as >far as I know) extremely difficult to set up a synchronised >environment across two or more PCs. rsync is my personal cheat. :) >For email, gmail, while not perfect, is a reasonably good client, and I can >use it on multiple PCs. Of course, IMAP allows this even without gmail. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From barry at python.org Tue Apr 1 17:38:25 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 1 Apr 2014 11:38:25 -0400 Subject: [Python-ideas] Browser for mailing lists References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> Message-ID: <20140401113825.3d8ae685@anarchist.wooz.org> On Apr 01, 2014, at 10:22 AM, Chris Angelico wrote: >Drifting off-topic, but I think this is actually a subtle feature. >I've often joined a mailing list to ask one specific question, and >then seen questions that I can help answer. In fact, that's how I came >to be stuck on python-list and family... of course, other people may >not see that as a positive feature... Another feature/wishlist of MM3 is a reputation service where, let's say you've engaged with three lists on python.org. Each time you've had to do a email address confirmation dance, and get your postings moderated. Once you've done that a couple of times across a few lists (configurable via admin knobs), you're deemed a member of the *site* in good standing. Now you can post to any list that opts in with no further confirmations. If you game the system to spam, an admin can lower your reputation and then all your posts would be moderated or discarded. I think a system like this would really make it very easy for users to have both long term engagements with some lists, and low-barrier short term drive-by engagements with others. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From solipsis at pitrou.net Tue Apr 1 18:00:51 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 1 Apr 2014 18:00:51 +0200 Subject: [Python-ideas] Default value for input References: <20140401115842.GR16526@ando> <20140401130147.GT16526@ando> Message-ID: <20140401180051.04bac426@fsol> On Wed, 2 Apr 2014 00:01:47 +1100 Steven D'Aprano wrote: > > Yes, I have work-arounds. They aren't what you call elegant. Your > suggestion, to wrap the default in square brackets: > > What is the fish of the day? [trout a la creme] For the record, I would spell it "trout ? la cr?me" or rather "truite ? la cr?me". (you might also want to add some shallots, or perhaps they are implied :-)) cheers Antoine. From steve at pearwood.info Tue Apr 1 18:16:53 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 Apr 2014 03:16:53 +1100 Subject: [Python-ideas] Default value for input In-Reply-To: References: <20140401115842.GR16526@ando> <20140401130147.GT16526@ando> Message-ID: <20140401161652.GU16526@ando> On Tue, Apr 01, 2014 at 03:21:40PM +0100, Paul Moore wrote: > Windows has much richer line editing capabilities than "plain old > backspace" - sufficiently so that I end up very frustrated when > working on Unix programs that don't have readline support, because > they are *so* limited (no arrow keys? seriously?). Sorry, I didn't intend to put down the Windows editing experience, it's not something I've had much experience with in Python. And you are right, Python on Linux without readline support is *painful*. > Ignoring that point, though, pre-filled edit buffers are *extremely* > uncommon on Windows, and I would imagine them being very surprising to > a user. I'm not sure I've ever seen them on Unix either, Pre-filled buffers are quite common in curses applications like the mail-client mutt or the editor nano. I'm hoping for something lightweight and cross-platform, which rules out curses. > though, so > maybe the target audience for this has specialised or unusual > expectations? Well, yes, they would be users of console apps who expect user *friendly* text interfaces instead of the user-hostile ones most developers write *wink* > Personally, I'm used to (and comfortable with) the > "Enter a value [12]: " convention where this means that pressing enter > without providing a value gives 12 (and I'd just implement that > manually with if val='': val = '12'). That's fine for the typical "[Y]/n" one letter inputs that are extremely common, and I don't have a problem with that. But for longer text, the inability to modify the default except by retyping the entire thing is a pretty major failure of the user interface. Imagine if the only way to get a default value in your GUI or web app's edit field was to put a label next to it: Leave the field blank to accept "Something" as the default. Type a thing here: _______________________________ I wouldn't accept that in my GUI apps, and I don't think that it's acceptable in console apps either. > I'm not sure how easy it would be to implement for Windows. I've never > needed it, so I've never looked into it. > > -1 on it being a builtin capability of input(). -0 on it being a > separate builtin or stdlib function (mainly because I don't think it's > needed often enough to be worth it). It seems to me it'd be fine as a > package on PyPI, or maybe a recipe. I obviously don't agree with you, but I appreciate the comments. Regardless of what happens in 3.5, I'll look at putting something up on PyPI, although chances are it will require readline support and won't be platform-independent. That will limit its use drastically. -- Steven From guido at python.org Tue Apr 1 18:30:12 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Apr 2014 09:30:12 -0700 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> Message-ID: Nice come-back! Responses inline. On Tue, Apr 1, 2014 at 6:26 AM, Nick Coghlan wrote: > On 31 March 2014 02:05, Guido van Rossum wrote: > > On Sat, Mar 29, 2014 at 7:17 PM, Nick Coghlan > wrote: > [...] > >> Background > >> ========== > >> > >> Over the course of Python 3's evolution, a number of adjustments have > been > >> made to the core ``bytes`` and ``bytearray`` types as additional > practical > >> experience was gained with using them in code beyond the Python 3 > standard > >> library and test suite. However, to date, these changes have been made > >> on a relatively ad hoc tactical basis as specific issues were > identified, > >> rather than as part of a systematic review of the APIs of these types. > > > > I'm not sure you can claim that. We probably have more information based > on > > experience now than when we did the redesign. (At that time most > experience > > was based on using str() for binary data.) > > Yeah, I was mostly thinking of the change to make the search APIs > accept both integers and subsequences when I wrote that. I'll try to > think of a better way of wording it. > That might have felt ad-hoc, but it was in line with the idea that bytes follow the patterns of both tuple and string. (And similar for bytearray.) > I also realised in reviewing the docs that a key part of the problem > may actually be a shortcut I took in the sequence docs rewrite that I > did quite a while ago now - the bytes/bytearray docs are currently > written in terms of *how they differ from str*. They don't really > cover the "container of integers" aspect particularly well. I now > believe a better way to tackle that would to be upfront that these > types basically have two APIs on a single class: their core tuple/list > of integers "arbitrary binary data" API, and then the str-inspired > "binary data with ASCII segments" API on top of that. > Right, and then you can follow up with details about how the APIs differ from their equivalents in tuple and string. I imagine there are three categories here (some may be empty): APIs that are the union of the corresponding tuple and string APIs; APIs that are like one of the "base" classes with some restrictions or extensions that can't be explained by referring to the other "base"; and APIs that are unique to bytes. > >> This > >> approach has allowed inconsistencies to creep into the API design as to > >> which > >> input types are accepted by different methods. Additional > inconsistencies > >> linger from an earlier pre-release design where there was *no* separate > >> ``bytearray`` type, and instead the core ``bytes`` type was mutable > (with > >> no immutable counterpart), as well as from the origins of these types in > >> the text-like behaviour of the Python 2 ``str`` type. > > > > You make it sound as if modeling bytes() after Python 2's str() was an > > accident. It wasn't. > > Sorry, didn't mean to imply that. More that we hadn't previously sat > down and thought through how best to clearly articulate this in the > docs, and hence some of the current inconsistencies hadn't become > clear. > Heh, I get defensive when you say bad things about the language. Not so much about the docs (too often I don't know what's in the docs myself because my knowledge predates the docs :-). > >> This PEP aims to provide the missing systematic review, with the goal of > >> ensuring that wherever feasible (given backwards compatibility > >> constraints) > >> these current inconsistencies are addressed for the Python 3.5 release. > > > > I would like to convince you to aim lower, drop the "systematic review", > and > > just focus on some changes that are likely to improve users' experience > > (which includes porting Python 2 code). > > After re-reading the current docs (which I also wrote, but this aspect > was a mere afterthought at the time), I'm thinking a useful course of > action in parallel with this PEP will be for me to work on improving > the Python 3.4 docs for these types. The bits I find too hard to > explain then become fodder for tweaks in 3.5. > I am very much in favor of this approach. Many API improvements I've made myself have come from attempts to write documentation, and I imagine I'm not unique. > >> Proposals > >> ========= > > >> * more consistently accepting length 1 ``bytes`` objects as input where > an > >> integer between ``0`` and ``255`` inclusive is expected, and > vice-versa > > > > > > Not sure I like this as a goal. OK, stronger: I don't like this goal. > > Roger. As noted above, I now think we can address this by splitting > the API documentation instead, so that there's a clear "tuple/list of > ints" section and a "binary data with ASCII segments" section. Some > hybrid APIs (like the search ones) may appear in both. In terms of > analogies to other types: > > Behaviour is common to tuple + str: hybrid API for bytes + bytearray > Behaviour is list-only: int-only API for bytearray > Behaviour is str-only: str-like only API for bytes + bytearray > Heh, I think I just accidentally reinvented that same categorization above. :-) > Now that I've framed the question that way, I think I can not only > make it make sense in the docs, but I believe the 3.4 behaviour is > already pretty close to consistent with it. > > The proposed bytes.byte() constructor would then more cleanly handle > the few cases where it may be desirable to pass an int to a str-like > API (such as replace()) > > >> * allowing users to easily convert integer output to a length 1 > ``bytes`` > >> object > > > > I think you meant integer values instead of output? > > Sort of - I was thinking of reversing the effects of indexing here. > That is, replacing the current: > > x = data[0:1] > > with: > > x = bytes.byte(data[0]) > Hm. I don't find that very attractive. You can't write Python 2/3 code using that idiom, and it's a lot longer than the original. The only redeeming feature is that it clearly fails when data is empty, and possibly that you don't have to compute the second index (which could be awkward if the first index is an expression). I'm not denying that we need bytes.byte(), but this doesn't sound like much of a motivation. Just pointing to the need of bytes/bytestring equivalents for chr() makes more sense to me. > > In Python 2 we did this > > with the global function chr(), but in Python 3 that creates a str(). > (The > > history of chr() and ord() as built-in functions is that they long > predates > > the notion of methods (class- or otherwise), and their naming comes > straight > > from Pascal.) > > > > Anyway, I don't know that the use case is so common that it needs more > than > > bytes([i]) or bytearray([i]) -- if there is an argument to be made for > > bytes.byte(i) and bytearray.byte(i) it would be that the [i] in the > > constructor is somewhat hard to grasp. > > Since I was mostly thinking about an alternative to slicing to convert > an index lookup back to a bytes object, this doesn't seem appealing to > me: > > x = bytes([data[0]]) > Fair enough. > The other one is that "bytes([i])" doesn't play nice with higher order > functions like map. > Also fair enough; having to define a helper function feels bad. All in all, I do think we need bytes.byte() and bytearray.byte(). We may just have to fine-tune the motivation a bit. :-) > I don't expect wanting this to be *hugely* common, but I do think > there's value in having the primitive conversion operation implied by > the constructor behaviour available as a Python level operation. > Yes. > >> Alternate Constructors > >> ---------------------- > >> > >> The ``bytes`` and ``bytearray`` constructors currently accept an integer > >> argument, but interpret it to mean a zero-filled object of the given > >> length. > > > > > > This is one of the two legacies of the original "mutable bytes" design, > and > > I agree we should strive to replace it -- although I think one round of > > deprecation may be too quick. > > Postponing removal to 3.7 or indefinitely is fine by me. > > While I think it should go away, I'm in no hurry to get rid of it - it > started bothering me less once I realised you can already safely call > bytes on arbitrary objects by passing them through memoryview first > (as that doesn't have the mutable legacy that causes problems with > integer input). > I'm not sure I quite see the use case. memoryview() doesn't take "arbitrary objects" -- it takes objects that implement the buffer protocol (if that's still the name :-). Are you saying that the advantage of going through memoryview() is that it fails fast when you accidentally pass it an integer(-like object)? [...] > >> For ``bytearray``, a ``from_len`` constructor is proposed that > >> preallocates > >> the buffer filled with a particular value (default to ``0``) as a direct > >> replacement for the current constructor behaviour, rather than having to > >> use > >> sequence repetition to achieve the same effect in a less intuitive way:: > >> > >> >>> bytearray.from_len(3) > >> bytearray(b'\x00\x00\x00') > >> >>> bytearray.from_len(3, 6) > >> bytearray(b'\x06\x06\x06') > >> > >> This part of the proposal was covered by an existing issue > >> [empty-buffer-issue]_ and a variety of names have been proposed > >> (``empty_buffer``, ``zeros``, ``zeroes``, ``allnull``, ``fill``). The > >> specific name currently proposed was chosen by analogy with > >> ``dict.fromkeys()`` and ``itertools.chain.from_iter()`` to be completely > >> explicit that it is an alternate constructor rather than an in-place > >> mutation, as well as how it differs from the standard constructor. > > > > I think you need to brainstorm more on the name; from_len() looks pretty > > awkward. And I think it's better to add it to bytes() as well, since the > two > > classes intentionally try to be as similar as possible. > > I initially liked Barry's "fill" suggestion, but then realised it read > too much like an in-place operation (at least to my mind). Here are > some examples (using Brett's suggestion of a keyword only second > parameter): > > bytearray.zeros(3) # NumPy spelling, no configurable fill value > bytearray.fill(3) > bytearray.fill(3, fillvalue=6) > bytearray.filled(3) > bytearray.filled(3, fillvalue=6) > > To be honest, I'm actually coming around to the "just copy the 'zeros' > name from NumPy and call it done" view on this one. I don't have a > concrete use case for a custom fill value, and I think I'll learn > quickly enough that it uses the shorter spelling. > +1 [...] > >> * ``bytes.byte()`` is defined above as accepting length 1 binary > sequences > >> as individual bytes, but this is currently inconsistent with the main > >> ``bytes`` constructor:: > >> > >> >>> bytes([b"a", b"b", b"c"]) > >> Traceback (most recent call last): > >> File "", line 1, in > >> TypeError: 'bytes' object cannot be interpreted as an integer > >> > >> Should the ``bytes`` constructor be changed to accept iterables of > >> length 1 > >> bytes objects in addition to iterables of integers? If so, should it > >> allow a mixture of the two in a single iterable? > > > > Noooooooooooooooooooooooooo!!!!! > > Yeah, it bothered me, too :) > > As you suggest, I think it makes sense to extrapolate this the other > way and change the definition of bytes.byte() to be a true inverse of > ord() for binary data. > +1 > >> Iteration > >> --------- > >> > >> Iteration over ``bytes`` objects and other binary sequences produces > >> integers. Rather than proposing a new method that would need to be added > >> not only to ``bytes``, ``bytearray`` and ``memoryview``, but potentially > >> to third party types as well, this PEP proposes that iteration to > produce > >> length 1 ``bytes`` objects instead be handled by combining ``map`` with > >> the new ``bytes.byte()`` alternate constructor proposed above:: > >> > >> for x in map(bytes.byte, data): > >> # x is a length 1 ``bytes`` object, rather than an integer > >> # This works with *any* container of integers in the range > >> # 0 to 255 inclusive > > > > I can see why you don't like a new method, but this idiom is way too > verbose > > and unintuitive to ever gain traction. Let's just add a new method to all > > three types, 3rd party types will get the message. > > Fair enough. Is "iterbytes()" OK as the name?: > > for x in date.iterbytes(): > # x is a length 1 ``bytes`` object, rather than an integer > # This works with *any* container of integers in the range > # 0 to 255 inclusive > +1 > >> Consistent support for different input types > >> -------------------------------------------- > >> > >> In Python 3.3, the binary search operations (``in``, ``count()``, > >> ``find()``, ``index()``, ``rfind()`` and ``rindex()``) were updated to > >> accept integers in the range 0 to 255 (inclusive) as their first > argument > >> (in addition to the existing support for binary sequences). > > > > > > I wonder if that wasn't a bit over-zealous. While 'in', count() and > index() > > are sequence methods (looking for elements) that have an extended meaning > > (looking for substrings) for string types, the find() and r*() variants > are > > only defined for strings. > > I suspect they're using the same underlying search code, although I > haven't actually checked. > OK, it's water under the bridge anyway. [...] > "replace()" seems like the only one where a reasonable case might be > made to allowing integer input (and that's actually the one Brandon > was asking about that got me thinking along these lines in the first > place). > I think not. It really works on substrings, length-one strings are just a common case. [...] > I'm not sure when I'll get the PEP updated (since this isn't an urgent > problem, I just wanted to get the initial draft of the PEP written > while the problem was fresh in my mind), but I think the end result > should be relatively non-controversial once I incorporate your > feedback. > No hurries. And you're welcome! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Apr 1 18:37:47 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Apr 2014 09:37:47 -0700 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401161652.GU16526@ando> References: <20140401115842.GR16526@ando> <20140401130147.GT16526@ando> <20140401161652.GU16526@ando> Message-ID: On Tue, Apr 1, 2014 at 9:16 AM, Steven D'Aprano wrote: > Leave the field blank to accept "Something" as the default. > Type a thing here: _______________________________ > > I wouldn't accept that in my GUI apps, and I don't think that it's > acceptable in console apps either. > Ah, but that's the difference. In GUI apps it's the standard approach (although it's amazing how many GUI apps tend to get edge cases wrong, and it doesn't combine well with the new trend to put dimmed "hint" text in the box instead). But in console apps it's *not* common, except perhaps in certain curses-based apps (which really are trying to emulate GUI apps using an older paradigm). I think it's fine for people to figure out how to do this using the available tools (e.g readline) and experiment. I don't think the idea is anywhere near the bar of adding to the stdlib. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Apr 1 18:42:10 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 1 Apr 2014 17:42:10 +0100 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401161652.GU16526@ando> References: <20140401115842.GR16526@ando> <20140401130147.GT16526@ando> <20140401161652.GU16526@ando> Message-ID: On 1 April 2014 17:16, Steven D'Aprano wrote: >> -1 on it being a builtin capability of input(). -0 on it being a >> separate builtin or stdlib function (mainly because I don't think it's >> needed often enough to be worth it). It seems to me it'd be fine as a >> package on PyPI, or maybe a recipe. > > I obviously don't agree with you, but I appreciate the comments. > Regardless of what happens in 3.5, I'll look at putting something up on > PyPI, although chances are it will require readline support and won't be > platform-independent. That will limit its use drastically. Having a better idea of your use case (lighter-weight version of a curses-style interface) I can see why it would be useful. Although I didn't think anyone still write text-based UIs ;-) With that in mind, and *particularly* since the curses module is not supported on Windows, I'm persuaded that it's a useful idea (to have available *somewhere*). But probably only if it's cross-platform, and even then I'd wouldn't expect it to be part of the built in input(). After all getpass is an even more common requirement for user input, and *that* didn't warrant being built into input :-) Paul From storchaka at gmail.com Tue Apr 1 19:14:17 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 01 Apr 2014 20:14:17 +0300 Subject: [Python-ideas] Default value for input In-Reply-To: References: <20140401115842.GR16526@ando> <20140401130147.GT16526@ando> Message-ID: 01.04.14 17:21, Paul Moore ???????(??): > -1 on it being a builtin capability of input(). -0 on it being a > separate builtin or stdlib function (mainly because I don't think it's > needed often enough to be worth it). It seems to me it'd be fine as a > package on PyPI, or maybe a recipe. Seconded. From ncoghlan at gmail.com Tue Apr 1 23:03:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 2 Apr 2014 07:03:24 +1000 Subject: [Python-ideas] Browser for mailing lists In-Reply-To: <20140401113825.3d8ae685@anarchist.wooz.org> References: <85d2h4ohdd.fsf@benfinney.id.au> <1396152153.12277.YahooMailNeo@web181001.mail.ne1.yahoo.com> <5339B929.1040000@stoneleaf.us> <20140401113825.3d8ae685@anarchist.wooz.org> Message-ID: On 2 Apr 2014 01:41, "Barry Warsaw" wrote: > > On Apr 01, 2014, at 10:22 AM, Chris Angelico wrote: > > >Drifting off-topic, but I think this is actually a subtle feature. > >I've often joined a mailing list to ask one specific question, and > >then seen questions that I can help answer. In fact, that's how I came > >to be stuck on python-list and family... of course, other people may > >not see that as a positive feature... > > Another feature/wishlist of MM3 is a reputation service where, let's say > you've engaged with three lists on python.org. Each time you've had to do a > email address confirmation dance, and get your postings moderated. Once > you've done that a couple of times across a few lists (configurable via admin > knobs), you're deemed a member of the *site* in good standing. Now you can > post to any list that opts in with no further confirmations. If you game the > system to spam, an admin can lower your reputation and then all your posts > would be moderated or discarded. > > I think a system like this would really make it very easy for users to have > both long term engagements with some lists, and low-barrier short term > drive-by engagements with others. Stack Exchange works like that - if you're a member in good standing of another Stack Exchange site, your reputation starts at 100 rather than zero. Cheers, Nick. > > -Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Apr 2 03:54:57 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 2 Apr 2014 12:54:57 +1100 Subject: [Python-ideas] Subequence search (was: Fixing the Python 3 bytes constructor) In-Reply-To: <1396359700.20750.101419865.55C1D916@webmail.messagingengine.com> References: <5336F9A6.2030006@stoneleaf.us> <1396359700.20750.101419865.55C1D916@webmail.messagingengine.com> Message-ID: <20140402015456.GB16466@ando> On Tue, Apr 01, 2014 at 09:41:40AM -0400, random832 at fastmail.us wrote: > I have a mostly unrelated question. Is there a _general_ subsequence > search function [that works on lists or even iterables], or is it only > available for str and bytes? There's nothing built-in, but here is a naive version: http://code.activestate.com/recipes/577850-search-sequences-for-sub-sequence/ -- Steven From ethan at stoneleaf.us Wed Apr 2 04:29:09 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 01 Apr 2014 19:29:09 -0700 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> Message-ID: <533B75F5.9040601@stoneleaf.us> On 04/01/2014 09:30 AM, Guido van Rossum wrote: >>> On Sat, Mar 29, 2014 at 7:17 PM, Nick Coghlan wrote: >>>> >>>> x = bytes.byte(data[0]) > > Hm. I don't find that very attractive. You can't write Python 2/3 code using that idiom, and it's a lot longer than the > original. The only redeeming feature is that it clearly fails when data is empty, and possibly that you don't have to > compute the second index (which could be awkward if the first index is an expression). > > I'm not denying that we need bytes.byte(), but this doesn't sound like much of a motivation. Just pointing to the need > of bytes/bytestring equivalents for chr() makes more sense to me. We already have ord() and chr() -- maybe we should just add byte(). -- ~Ethan~ From ericsnowcurrently at gmail.com Wed Apr 2 06:40:59 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 1 Apr 2014 22:40:59 -0600 Subject: [Python-ideas] A namedtuple literal. Message-ID: (Inspired by https://mail.python.org/pipermail/python-ideas/2010-October/008532.html) (color: c, position: p) This would automatically created a namedtuple class and provide an instance of it. -eric p.s. Don't take this too seriously. :) From ericsnowcurrently at gmail.com Wed Apr 2 06:55:11 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 1 Apr 2014 22:55:11 -0600 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: On Tue, Apr 1, 2014 at 10:40 PM, Eric Snow wrote: > (Inspired by https://mail.python.org/pipermail/python-ideas/2010-October/008532.html) > > (color: c, position: p) > > This would automatically created a namedtuple class and provide an > instance of it. Ooh. How about an OrderedDict literal: ['a' : 1, 'b' : 2, 'c' : 3] (courtesy of https://mail.python.org/pipermail/python-ideas/2010-October/008484.html) reading-too-many-archived-threads-ly, -eric From haoyi.sg at gmail.com Wed Apr 2 06:59:40 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Tue, 1 Apr 2014 21:59:40 -0700 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: I personally made my own "namedtuple" "literal" class Container: pass def t(**kw): c = Container() for k, w in kw.items(): setattr(c, k, w) return c t(color=c, position=p) On Tue, Apr 1, 2014 at 9:55 PM, Eric Snow wrote: > On Tue, Apr 1, 2014 at 10:40 PM, Eric Snow > wrote: > > (Inspired by > https://mail.python.org/pipermail/python-ideas/2010-October/008532.html) > > > > (color: c, position: p) > > > > This would automatically created a namedtuple class and provide an > > instance of it. > > Ooh. How about an OrderedDict literal: > > ['a' : 1, 'b' : 2, 'c' : 3] > > (courtesy of > https://mail.python.org/pipermail/python-ideas/2010-October/008484.html) > > reading-too-many-archived-threads-ly, > > -eric > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Apr 2 07:07:03 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 1 Apr 2014 23:07:03 -0600 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: On Tue, Apr 1, 2014 at 10:59 PM, Haoyi Li wrote: > I personally made my own "namedtuple" "literal" > > class Container: > pass > > def t(**kw): > c = Container() > for k, w in kw.items(): > setattr(c, k, w) > return c > > t(color=c, position=p) Nice! Alas, you lose your kw order so it's less namedtuply, which is precisely what I've been digging the list archives about. :) -eric From flying-sheep at web.de Wed Apr 2 08:54:18 2014 From: flying-sheep at web.de (Philipp A.) Date: Wed, 2 Apr 2014 08:54:18 +0200 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: 2014-04-02 7:07 GMT+02:00 Eric Snow : > Nice! Alas, you lose your kw order so it's less namedtuply, which is > precisely what I've been digging the list archives about. :) > > -eric > i have this one floating around: https://gist.github.com/flying-sheep/9929117 it can easily be adapted for namedtuples. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Apr 2 13:40:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 2 Apr 2014 21:40:51 +1000 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: <533B75F5.9040601@stoneleaf.us> References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> Message-ID: On 2 Apr 2014 12:52, "Ethan Furman" wrote: > > On 04/01/2014 09:30 AM, Guido van Rossum wrote: > >>>> On Sat, Mar 29, 2014 at 7:17 PM, Nick Coghlan wrote: >>>>> >>>>> >>>>> x = bytes.byte(data[0]) >> >> >> Hm. I don't find that very attractive. You can't write Python 2/3 code using that idiom, and it's a lot longer than the >> original. The only redeeming feature is that it clearly fails when data is empty, and possibly that you don't have to >> compute the second index (which could be awkward if the first index is an expression). >> >> I'm not denying that we need bytes.byte(), but this doesn't sound like much of a motivation. Just pointing to the need >> of bytes/bytestring equivalents for chr() makes more sense to me. > > > We already have ord() and chr() -- maybe we should just add byte(). I thought of that, but it seems like a recipe for typos and confusion. bytes.byte and bytearray.byte seem clearer and safer. Cheers, Nick. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Apr 2 14:01:03 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 2 Apr 2014 08:01:03 -0400 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> Message-ID: <10A34668-213D-450D-A00C-9F07CC33D03D@stufft.io> On Apr 2, 2014, at 7:40 AM, Nick Coghlan wrote: > > On 2 Apr 2014 12:52, "Ethan Furman" wrote: > > > > On 04/01/2014 09:30 AM, Guido van Rossum wrote: > > > >>>> On Sat, Mar 29, 2014 at 7:17 PM, Nick Coghlan wrote: > >>>>> > >>>>> > >>>>> x = bytes.byte(data[0]) > >> > >> > >> Hm. I don't find that very attractive. You can't write Python 2/3 code using that idiom, and it's a lot longer than the > >> original. The only redeeming feature is that it clearly fails when data is empty, and possibly that you don't have to > >> compute the second index (which could be awkward if the first index is an expression). > >> > >> I'm not denying that we need bytes.byte(), but this doesn't sound like much of a motivation. Just pointing to the need > >> of bytes/bytestring equivalents for chr() makes more sense to me. > > > > > > We already have ord() and chr() -- maybe we should just add byte(). > > I thought of that, but it seems like a recipe for typos and confusion. bytes.byte and bytearray.byte seem clearer and safer. > > Cheers, > Nick. > > I don?t like byte(), way to much potential for confusion with bytes(), but maybe bchr() is a reasonable thing. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Wed Apr 2 14:16:28 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 2 Apr 2014 22:16:28 +1000 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: <10A34668-213D-450D-A00C-9F07CC33D03D@stufft.io> References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> <10A34668-213D-450D-A00C-9F07CC33D03D@stufft.io> Message-ID: On 2 Apr 2014 22:01, "Donald Stufft" wrote: > > > On Apr 2, 2014, at 7:40 AM, Nick Coghlan wrote: > >> >> On 2 Apr 2014 12:52, "Ethan Furman" wrote: >> > >> > On 04/01/2014 09:30 AM, Guido van Rossum wrote: >> > >> >>>> On Sat, Mar 29, 2014 at 7:17 PM, Nick Coghlan wrote: >> >>>>> >> >>>>> >> >>>>> x = bytes.byte(data[0]) >> >> >> >> >> >> Hm. I don't find that very attractive. You can't write Python 2/3 code using that idiom, and it's a lot longer than the >> >> original. The only redeeming feature is that it clearly fails when data is empty, and possibly that you don't have to >> >> compute the second index (which could be awkward if the first index is an expression). >> >> >> >> I'm not denying that we need bytes.byte(), but this doesn't sound like much of a motivation. Just pointing to the need >> >> of bytes/bytestring equivalents for chr() makes more sense to me. >> > >> > >> > We already have ord() and chr() -- maybe we should just add byte(). >> >> I thought of that, but it seems like a recipe for typos and confusion. bytes.byte and bytearray.byte seem clearer and safer. >> >> Cheers, >> Nick. >> >> > > I don't like byte(), way to much potential for confusion with bytes(), but maybe bchr() is a reasonable thing. There's no need for it to be a builtin at all. The class method alternative constructor approach handles the problem just fine. Cheers, Nick. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Apr 2 14:17:29 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 2 Apr 2014 08:17:29 -0400 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> <10A34668-213D-450D-A00C-9F07CC33D03D@stufft.io> Message-ID: <3CD0231F-A691-4A80-9EBC-83684EC13054@stufft.io> On Apr 2, 2014, at 8:16 AM, Nick Coghlan wrote: > > On 2 Apr 2014 22:01, "Donald Stufft" wrote: > > > > > > On Apr 2, 2014, at 7:40 AM, Nick Coghlan wrote: > > > >> > >> On 2 Apr 2014 12:52, "Ethan Furman" wrote: > >> > > >> > On 04/01/2014 09:30 AM, Guido van Rossum wrote: > >> > > >> >>>> On Sat, Mar 29, 2014 at 7:17 PM, Nick Coghlan wrote: > >> >>>>> > >> >>>>> > >> >>>>> x = bytes.byte(data[0]) > >> >> > >> >> > >> >> Hm. I don't find that very attractive. You can't write Python 2/3 code using that idiom, and it's a lot longer than the > >> >> original. The only redeeming feature is that it clearly fails when data is empty, and possibly that you don't have to > >> >> compute the second index (which could be awkward if the first index is an expression). > >> >> > >> >> I'm not denying that we need bytes.byte(), but this doesn't sound like much of a motivation. Just pointing to the need > >> >> of bytes/bytestring equivalents for chr() makes more sense to me. > >> > > >> > > >> > We already have ord() and chr() -- maybe we should just add byte(). > >> > >> I thought of that, but it seems like a recipe for typos and confusion. bytes.byte and bytearray.byte seem clearer and safer. > >> > >> Cheers, > >> Nick. > >> > >> > > > > I don?t like byte(), way to much potential for confusion with bytes(), but maybe bchr() is a reasonable thing. > > There's no need for it to be a builtin at all. The class method alternative constructor approach handles the problem just fine. > > Cheers, > Nick > > I don?t much care which way it goes, I was just mentioning it as an alternative :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From yselivanov.ml at gmail.com Wed Apr 2 17:20:18 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 02 Apr 2014 11:20:18 -0400 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: <533C2AB2.4080905@gmail.com> On 2014-04-02, 12:40 AM, Eric Snow wrote: > (Inspired by https://mail.python.org/pipermail/python-ideas/2010-October/008532.html) > > (color: c, position: p) While it looks extremely neat, there is one design flaw that is shared with JavaScript's object literals. In JS, if you want to define a simple object, you can write it in two ways: {'spam': 'ham'} or {spam: 'ham'} But when you have a key name defined in a variable, you'll need to do key = 'spam' o = {} o[key] = 'ham' Where in Python, you'd simply write {key: 'ham'}. So for Python, I think that having unquoted keys in literals is a bad idea. Yury From flying-sheep at web.de Wed Apr 2 18:34:00 2014 From: flying-sheep at web.de (Philipp A.) Date: Wed, 2 Apr 2014 18:34:00 +0200 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <533C2AB2.4080905@gmail.com> References: <533C2AB2.4080905@gmail.com> Message-ID: 2014-04-02 17:20 GMT+02:00 Yury Selivanov : But when you have a key name defined in a variable, you?ll need to do key = 'spam' o = {} o[key] = 'ham' Where in Python, you?d simply write {key: ?ham?}. So for Python, I think that having unquoted keys in literals is a bad idea. i totally agree for dicts, but here i don?t. instead i?d require unquoted keys like in kwargs. 1. the key here needs to be a valid identifier to be accessed using namedtuple.key_name. this could be ensured by the parser if only unquoted keys are allowed, not if string keys or even variable keys containing arbitrary objects are allowed 2. i don?t actually want this to be something with dynamic keys. it?s supposed to be a namedtuple, and namedtuples are static as well regarding their keys. i think it?s best to encourage that functions returning namedtuples only return one kind of them, so that you can ensure that returnvalue.my_keyworks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Apr 2 18:42:32 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 02 Apr 2014 12:42:32 -0400 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: <533C2AB2.4080905@gmail.com> Message-ID: <533C3DF8.5090007@gmail.com> On 2014-04-02, 12:34 PM, Philipp A. wrote: > 2014-04-02 17:20 GMT+02:00 Yury Selivanov : > > But when you have a key name defined in a variable, you?ll need to do > > key = 'spam' > o = {} > o[key] = 'ham' > > Where in Python, you?d simply write {key: ?ham?}. > > So for Python, I think that having unquoted keys in literals is a bad idea. > > i totally agree for dicts, but here i don?t. instead i?d require unquoted > keys like in kwargs. > > 1. > > the key here needs to be a valid identifier to be accessed using > namedtuple.key_name. > > this could be ensured by the parser if only unquoted keys are allowed, > not if string keys or even variable keys containing arbitrary objects are > allowed > 2. > > i don?t actually want this to be something with dynamic keys. > > it?s supposed to be a namedtuple, and namedtuples are static as well > regarding their keys. > > i think it?s best to encourage that functions returning namedtuples only > return one kind of them, so that you can ensure that returnvalue.my_keyworks. > All good points. Generally, I'm +1 on the idea. Yury From steve at pearwood.info Wed Apr 2 23:27:00 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 3 Apr 2014 08:27:00 +1100 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: <533C2AB2.4080905@gmail.com> Message-ID: <20140402212656.GA25514@ando> On Wed, Apr 02, 2014 at 06:34:00PM +0200, Philipp A. wrote: > 2014-04-02 17:20 GMT+02:00 Yury Selivanov : > > > But when you have a key name defined in a variable, you?ll need to do > > > > key = 'spam' > > o = {} > > o[key] = 'ham' > > > > Where in Python, you?d simply write {key: ?ham?}. > > > > So for Python, I think that having unquoted keys in literals is a > > bad idea. > > i totally agree for dicts, but here i don?t. instead i?d require unquoted > keys like in kwargs. These would be reasonable points to consider if the idea of literal namedtuple syntax was a serious idea to consider, but I don't think it is. Please look again at Eric's original post, and take note of the last thing he says, and remember that there is a tradition in Anglo-American culture of making "April Fools" jokes and other non-serious suggestions. namedtuple, as it exists, creates a class, which you then instantiate. It's a factory function, not a class itself: py> from collections import namedtuple py> type(namedtuple) But Eric's suggestion skips creating a distinct class and gives you an instance straight away. So what class will the instance belong to? There are three obvious possibilities, and they're all unfortunate: (1) every instance is a singleton of a unique class; (2) there is a single NamedTuple class that every instance belongs to; (3) the namedtuple literal has to cache the keys it has seen, and return the same class when given the same keys in the same order. None of them are ideal; #3 means that if module a creates a namedtuple, and module b creates a namedtuple that just happens to use the same field names, they will have the same type. Most of the time that will not matter, but occasionally it will lead to problems. (e.g. suppose module a enforces some constraint on the field values, but module b does not. Then if a namedtuple from b somehow gets passed to module a, which does an isinstance type check to validate the namedtuple, the constraint will be broken). #2 on the other hand means that: vector = (x: 2.3, y: -4.2, z: 1.7) colour = (red: 100, green: 5, blue: 203) will be considered the same type. For something meant to be considered as a record or struct, I think that's a bad idea. #1 on the other hand means that: p = (x: 2, y: 3) q = (x: 2, y: 3) will be considered different types, which is surely going to come to a big surprise to most people. Of the three scenarios, I think #3 is the least worst. But I don't really like it. > 2. > > i don?t actually want this to be something with dynamic keys. > > it?s supposed to be a namedtuple, and namedtuples are static as well > regarding their keys. I'm not entirely sure what you mean by this. The namedtuple factory function accepts dynamic keys, not just literals: py> from collections import namedtuple py> name = 'spam' py> K = namedtuple('K', (name, 'eggs', 'cheese')) py> K._fields ('spam', 'eggs', 'cheese') I don't think literal syntax should be any more restrictive: name = "spam" t = (name: 1, 'eggs': 2, 'cheese': 3) assert t._fields == ('spam', 'eggs', 'cheese') assert getattr(t, name) == 1 But now I'm also guilty of ignoring Eric's warning not to take the suggestion too seriously. -- Steven From yselivanov.ml at gmail.com Wed Apr 2 23:51:50 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 02 Apr 2014 17:51:50 -0400 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <20140402212656.GA25514@ando> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> Message-ID: <533C8676.2000504@gmail.com> On 2014-04-02, 5:27 PM, Steven D'Aprano wrote: > On Wed, Apr 02, 2014 at 06:34:00PM +0200, Philipp A. wrote: >> 2014-04-02 17:20 GMT+02:00 Yury Selivanov : >> >>> But when you have a key name defined in a variable, you?ll need to do >>> >>> key = 'spam' >>> o = {} >>> o[key] = 'ham' >>> >>> Where in Python, you?d simply write {key: ?ham?}. >>> >>> So for Python, I think that having unquoted keys in literals is a >>> bad idea. >> i totally agree for dicts, but here i don?t. instead i?d require unquoted >> keys like in kwargs. > These would be reasonable points to consider if the idea of literal > namedtuple syntax was a serious idea to consider, but I don't think it > is. Please look again at Eric's original post, and take note of the last > thing he says, and remember that there is a tradition in Anglo-American > culture of making "April Fools" jokes and other non-serious suggestions. Well, even if it was intended as an April Fools joke, I still like the idea :) > namedtuple, as it exists, creates a class, which you then instantiate. > It's a factory function, not a class itself: > > py> from collections import namedtuple > py> type(namedtuple) > Do you want dict literal to create a new class for each key combination? > But Eric's suggestion skips creating a distinct class and gives you an > instance straight away. So what class will the instance belong to? There > are three obvious possibilities, and they're all unfortunate: > > > (1) every instance is a singleton of a unique class; > > (2) there is a single NamedTuple class that every instance belongs to; I think it's going to be (2). Like what we currently have for regular 'tuple'. > (3) the namedtuple literal has to cache the keys it has seen, and return > the same class when given the same keys in the same order. > > > None of them are ideal; #3 means that if module a creates a namedtuple, > and module b creates a namedtuple that just happens to use the same > field names, they will have the same type. Most of the time that will > not matter, but occasionally it will lead to problems. I was told on this list once: namedtuple is a protocol. We don't need a common base class for them, we don't really need isinstance to work on them. > (e.g. suppose module a enforces some constraint on the field values, but > module b does not. Then if a namedtuple from b somehow gets passed to > module a, which does an isinstance type check to validate the > namedtuple, the constraint will be broken). > > #2 on the other hand means that: > > vector = (x: 2.3, y: -4.2, z: 1.7) > colour = (red: 100, green: 5, blue: 203) > > will be considered the same type. For something meant to be considered > as a record or struct, I think that's a bad idea. Same as (1, 2) is the same type as ('1', '2', '3'). > #1 on the other hand means that: > > p = (x: 2, y: 3) > q = (x: 2, y: 3) > > will be considered different types, which is surely going to come to a > big surprise to most people. But p will be equal to q. That's all that matters. Again, same for p = (2, 3) q = (2, 3) I think that if we indeed treat namedtuples as more convenient version of regular tuples, then Eric's proposal makes a lot of sense. Yury From abarnert at yahoo.com Wed Apr 2 23:59:23 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 2 Apr 2014 14:59:23 -0700 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <20140402212656.GA25514@ando> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> Message-ID: <7A6C12B1-A2A5-402F-A4D4-99D523B1E5E7@yahoo.com> On Apr 2, 2014, at 14:27, Steven D'Aprano wrote: > On Wed, Apr 02, 2014 at 06:34:00PM +0200, Philipp A. wrote: >> 2014-04-02 17:20 GMT+02:00 Yury Selivanov : >> >>> So for Python, I think that having unquoted keys in literals is a >>> bad idea. >> >> i totally agree for dicts, but here i don?t. instead i?d require unquoted >> keys like in kwargs. > > These would be reasonable points to consider if the idea of literal > namedtuple syntax was a serious idea to consider, but I don't think it > is. Please look again at Eric's original post, and take note of the last > thing he says, and remember that there is a tradition in Anglo-American > culture of making "April Fools" jokes and other non-serious suggestions. > > namedtuple, as it exists, creates a class, which you then instantiate. > It's a factory function, not a class itself: > > py> from collections import namedtuple > py> type(namedtuple) > > > > But Eric's suggestion skips creating a distinct class and gives you an > instance straight away. So what class will the instance belong to? There > are three obvious possibilities, and they're all unfortunate: > > > (1) every instance is a singleton of a unique class; > > (2) there is a single NamedTuple class that every instance belongs to; > > (3) the namedtuple literal has to cache the keys it has seen, and return > the same class when given the same keys in the same order. You're neglecting the fact namedtuple classes also have to have a "typename" argument. So clearly this syntax would already have to somehow figure out the right typename based on the keys. Once you solve that simple problem, all matching namedtuple literals in the same namespace are automatically the same type, with no caching needed. (But this also gives you the flexibility to explicitly rebind the name between two literals to create two different types when you need to, without having to reassign colour.__class__, which solves a very common problem.) And you're also neglecting the fact that we have more than just the keys, we also have the types of the values, and indeed even the values themselves. So, obviously, whenever the compiler sees this: colour = (red: 100, green: 5, blue: 203) ... it can tell that you're creating an IntRGB8Color[1], not a FloatRGBColor, or even an IntRGB16Color[2]. So, in conclusion, once you solve the problem of figuring out the intended semantic meaning of the namedtuple literal, everything else is trivial. So dismissing this suggestion as a joke just because it was posted on April Fool's Day seems unfair. [1] Of course you wanted it to create an IntRGB8Colour, but you can't expect Python to magically guess whether you want US or UK spellings; that would just be silly. [2] In theory these are legal values for an IntRGB16Color, but really, how often do you need literals that are indistinguishable from black but not actually black? I don't think we need to handle that edge case. Perl may have many ways to represent colors that look completely black, but in Python, (0, 0, 0) is the one and only one obvious way to do it. From ethan at stoneleaf.us Thu Apr 3 00:22:55 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 02 Apr 2014 15:22:55 -0700 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <7A6C12B1-A2A5-402F-A4D4-99D523B1E5E7@yahoo.com> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> <7A6C12B1-A2A5-402F-A4D4-99D523B1E5E7@yahoo.com> Message-ID: <533C8DBF.5080305@stoneleaf.us> On 04/02/2014 02:59 PM, Andrew Barnert wrote: > > So dismissing this suggestion as a joke just because it was posted on > April Fool's Day seems unfair. It's being dismissed as a joke because the OP said to. -- ~Ethan~ From greg.ewing at canterbury.ac.nz Thu Apr 3 04:05:20 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Apr 2014 15:05:20 +1300 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <7A6C12B1-A2A5-402F-A4D4-99D523B1E5E7@yahoo.com> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> <7A6C12B1-A2A5-402F-A4D4-99D523B1E5E7@yahoo.com> Message-ID: <533CC1E0.8030408@canterbury.ac.nz> On 03/04/14 10:59, Andrew Barnert wrote: > [2] In theory these are legal values for an IntRGB16Color, but really, how > often do you need literals that are indistinguishable from black but not > actually black? This wouldn't meet the exacting needs of the Assassins' Guild of Ankh-Morporkh, who are very fashion-conscious and make quite fine distinctions between the shades of black they wear. -- Greg From ericsnowcurrently at gmail.com Thu Apr 3 04:58:22 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 2 Apr 2014 20:58:22 -0600 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <20140402212656.GA25514@ando> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> Message-ID: On Wed, Apr 2, 2014 at 3:27 PM, Steven D'Aprano wrote: > These would be reasonable points to consider if the idea of literal > namedtuple syntax was a serious idea to consider, but I don't think it > is. Please look again at Eric's original post, and take note of the last > thing he says, and remember that there is a tradition in Anglo-American > culture of making "April Fools" jokes and other non-serious suggestions. Yep. I was digging through list archives and stumbled upon this idea several times (both for namedtuple and for OrderedDict). It's not a terrible idea, but definitely overkill, IMHO. In the one case where Guido gave his opinion, he gave a -100. :) I figured I'd throw it out there for April 1st. I guess without more context the idea is less obviously bad. Maybe I should have just picked a stinkier idea. -eric From rosuav at gmail.com Thu Apr 3 05:12:18 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 3 Apr 2014 14:12:18 +1100 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> Message-ID: On Thu, Apr 3, 2014 at 1:58 PM, Eric Snow wrote: > I figured I'd throw it out there for April 1st. I guess without more > context the idea is less obviously bad. Maybe I should have just > picked a stinkier idea. It's pretty hard to come up with an idea so bad that it's obviously a joke. When I was knocking together the except-expression syntaxes, one of the four most-recommended notations actually started as a joke ("expr except Exception pass default"), and ended up actually being reasonably plausible. ChrisA From abarnert at yahoo.com Thu Apr 3 06:49:12 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 2 Apr 2014 21:49:12 -0700 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <533CC1E0.8030408@canterbury.ac.nz> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> <7A6C12B1-A2A5-402F-A4D4-99D523B1E5E7@yahoo.com> <533CC1E0.8030408@canterbury.ac.nz> Message-ID: On Apr 2, 2014, at 19:05, Greg Ewing wrote: > On 03/04/14 10:59, Andrew Barnert wrote: >> [2] In theory these are legal values for an IntRGB16Color, but really, how >> often do you need literals that are indistinguishable from black but not >> actually black? > > This wouldn't meet the exacting needs of the Assassins' Guild of > Ankh-Morporkh, who are very fashion-conscious and make quite fine > distinctions between the shades of black they wear. Python 3.x doesn't support Druidic Business Megalithics or any other Big Stone anymore, and with no OutOfCheese error in the stdlib the port to Hex is stalled indefinitely, so you can't seriously be suggesting that this use case matters. From ericsnowcurrently at gmail.com Thu Apr 3 08:39:32 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Apr 2014 00:39:32 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> Message-ID: On Thu, Mar 20, 2014 at 10:28 AM, Andrew Barnert wrote: > And as far as I can see, there is no way to solve this problem. The only way to add keyword order information without breaking all existing forwarding functions is to make the **kwargs parameter always preserve order (and make the **kwargs calling syntax preserve order if present)--that is, to make it always an OrderedDict, which Guido has already rejected. It's not all that bleak. Either of the options I outlined should still work, though option #1 (using a decorator) is better for the pass-through case that has come up (which I'm not convinced is such a big deal). I should have a rough PEP up in a day or two. Then I'll get my C OrderedDict applying cleanly and put up an implementation for the PEP, neither of which *should* take a lot of work. For what it's worth, that C implementation isn't all that fancy nor is it very optimized, but it still performs pretty well (last time I checked it was mostly the same as dict with the worst operations taking 4 times longer, which is much better than the pure Python version). This discussion has gotten me thinking about a related point (loosening the restriction on ** unpacking), but I'll start another thread on that one. -eric From ericsnowcurrently at gmail.com Thu Apr 3 08:54:35 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Apr 2014 00:54:35 -0600 Subject: [Python-ideas] loosening the restriction on what types may be unpacked using the ** syntax Message-ID: The ** keyword arg unpacking syntax in function calls is useful. However, currently there is an explicit check for a mapping type (presumably PyMapping_Check). So you can pass an object to dict(), but that same object *may* not work for keyword arg unpacking. For example: >>> from collections import OrderedDict >>> def f(**kwargs): return kwargs ... >>> kwargs = ((chr(i), i) for i in range(65, 68)) >>> f(**kwargs) Traceback (most recent call last): File "", line 1, in TypeError: f() argument after ** must be a mapping, not generator >>> f(**OrderedDict(kwargs)) {'B': 66, 'A': 65, 'C': 67} I'd like to relax this restriction so that anything you can pass to dict is also valid for keyword arg unpacking. This would mean you don't have to first create a dict to wrap the object before passing it in. Thoughts? -eric From abarnert at yahoo.com Thu Apr 3 10:55:41 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 01:55:41 -0700 (PDT) Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> Message-ID: <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> On Apr 2, 2014, at 23:39, Eric Snow wrote: > On Thu, Mar 20, 2014 at 10:28 AM, Andrew Barnert wrote: >> And as far as I can see, there is no way to solve this problem. The only way to add keyword order information without breaking all existing forwarding functions is to make the **kwargs parameter always preserve order (and make the **kwargs calling syntax preserve order if present)--that is, to make it always an OrderedDict, which Guido has already rejected. > > It's not all that bleak.? Either of the options I outlined should > still work, though option #1 (using a decorator) is better for the > pass-through case that has come up (which I'm not convinced is such a > big deal). It's definitely a big deal. In Python 3.4, it is trivial to write a wrapper that perfectly forwards its arguments. Or that modifies its arguments in somewhat, then perfectly forwards the modified version.?This is critical to being able to write decorators. Nearly every existing decorator in the stdlib, third-party libs, ActiveState recipes, and even your own projects does this. I'll show a dead-simple example below. But with either of your options, that would no longer be true. It would be at least a little more difficult to write a wrapper that perfectly forwards its arguments, and every existing decorator in the world would have to be rewritten to do so. Let's take your specific example, and simplify it even further.?I'll show why it doesn't _obviously_ work, and then you try to explain how you could make it work anyway: ? ? @preserve_kwargs_order ? ? def spam(**kwargs): ? ? ? ? print(kwargs) ? ? spam(a=1, b=2) So far, so good; your proposal means spam is guaranteed to get an OrderedDict with a before b. Now let's take a simple and useful general-purpose wrapper: ? ? def memoize(func): ? ? ? ? cache = {} ? ? ? ? def wrapper(*args, **kwargs): ? ? ? ? ? ? if (args, kwargs) not in _cache: ? ? ? ? ? ? ? ? cache[args, kwargs] = func(*args, **kwargs) ? ? ? ? ? ? return cache[args, kwargs] ? ? ? ? return wrapper ? ? eggs = memoize(spam) ? ? eggs(a=1, b=2) Here, spam is guaranteed to get an OrderedDict,?but it's arbitrary whether a comes before or after b. Why? Well, inside eggs (aka wrapper), kwargs is a plain old dict. So when it calls spam (aka func) with **kwargs, they get sent in whatever arbitrary order they had in the dict. Would anyone ever write or use a wrapper like this? Well, the Python developers thought it was useful enough to add a more complicated version of the same thing to the stdlib as functools.lru_cache, and there were dozens of recipes and third-party modules doing the same thing before it was added to the stdlib, so, yes.?And, more importantly, if you work through any other general-purpose wrapper, it's going to have the exact same problems. (It's marginally harder to work through, but maybe more obvious once you do so, with wrappers that modify their arguments, like functools.partial.) Can Python fix this for you magically? There are four potential places it could, and none of them will work.?When compiling memoize, Python obviously has no idea whether the value (or values) you will later pass to it are order-preserving functions or normal functions. At?runtime, when executing that compiled memoize definition, you still have the same problems. Later, when calling memoize on spam, Python now knows?that the value is an order-preserving function?but it no longer knows that memoize is intended to be a perfect-forwarding wrapper-generator. (In fact,?there is nothing special about the definition of memoize in the first place that makes it detectably a perfect-forwarding wrapper at any point. If that isn't obvious for memoize, consider?functools.lru_cache or functools.partial.) And finally, when calling the memoized function eggs, it's obviously still too late.So, there is no place that Python could automatically figure out that it needs to push the order-preservation upward from spam to eggs. The only way to make eggs end up as order-preserving is to change Python so that all functions (or at least all functions defined inside other functions) are?order-preserving. Either you or someone else suggested that if Python could just pass the kwargs dict straight through instead of?unpacking and repacking it, that would help. But it wouldn't. There's nothing marking eggs as an order-preserving function, so its kwargs is just a plain dict, and?passing that straight through can't help anything.?And besides, that idea isn't possible, or even (in general) sensible. Consider eggs=partial(spam, c=3), where the kwargs in eggs only has a and b, but the one inside spam has a, b, and c; obviously they can't be the same mapping. Or consider your original example (with explicit a and b params on spam before **kwargs) with memoize, where the kwargs in eggs has a, b, c, and d, but the one inside spam has only c and d. Now, obviously you could fix any particular wrapper. Presumably your magic decorator works by doing something detectable from Python, like a new co_flags flag. So, we could define two new functions (presumably in inspect and functools, respectively): ? ? def preserves_kwargs_order(func): ? ? ? ? return func.__code__.co_flags & 16 ? ? def wrap_preserve_kwargs_order(func): ? ? ? ? def wrapper(inner): ? ? ? ? ? ? if inspect.preserves_kwargs_order(func): ? ? ? ? ? ? ? ? return preserve_kwargs_order(inner) ? ? ? ? ? ? else: ? ? ? ? ? ? ? ? return inner ? ? ? ? return wrapper And now, all you have to do is add one line to most decorators and they're once again perfect forwarders: ? ? def memoize(func): ? ? ? ? _cache = {} ? ? ? ? @functools.wrap_preserve_kwargs_order(func) ? ? ? ? def wrapper(*args, **kwargs): ? ? ? ? ? ? if (args, kwargs) not in _cache: ? ? ? ? ? ? ? ? _cache[args, kwargs] = func(*args, **kwargs) ? ? ? ? ? ? return?_cache[args, kwargs] ? ? ? ? return wrapper But the point is, you still have to add that decorator to every wrapper function in the world or they're no longer perfect forwarders. You could even add that wrap_preserve_kwargs_order call into functools.wraps, and that would fix _many_ decorators (because many decorators are written to use functools.wraps), but still not all.?Most notably, the wrappers inside functools don't use wraps (even if you don't get the C-accelerated versions). So, that would still means that you have to add the new decorator to every wrapper function in the world that doesn't use wraps. From mistersheik at gmail.com Thu Apr 3 11:39:52 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 3 Apr 2014 02:39:52 -0700 (PDT) Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: <20140328105027.GH16526@ando> References: <20140328105027.GH16526@ando> Message-ID: I guess neither of you uses numpy? :) Please not "zeroes". Numpy uses "zeros". If it's going to be "allnull" please put the underscore in to be consistent with PEP 8: "all_null". Best, Neil On Friday, March 28, 2014 6:50:28 AM UTC-4, Steven D'Aprano wrote: > > On Fri, Mar 28, 2014 at 08:27:33PM +1000, Nick Coghlan wrote: > > One of the current annoyances with the bytes type in Python 3 is the > > way the constructor handles integers: > [...] > > > 1. Add "bytes.chr" such that "bytes.chr(x)" is equivalent to the PEP > > 361 defined "b'%c' % x" > > +1 on the concept, but please not "chr". How about bytes.byte(x)? That > emphasises that you are dealing with a single byte, and avoids > conflating chars (strings of length 1) with bytes. > > > > 2. Add "bytearray.allnull" and "bytes.allnull" such that > > "bytearray.allnull(x)" is equivalent to the current "bytearray(x)" int > > handling > > +1 on bytearray.allnull, with a mild preference for spelling it "zeroes" > instead. > > I'm indifferent about bytes.allnull. I'm not sure why I would use it > in preference to b'\0'*x, or for that matter why I would use it at all. > I suppose if there's a bytearray.allnul, for symmetry there should be a > bytes.allnul as well. > > > > 3. Deprecate the current "bytes(x)" and "bytearray(x)" int handling as > > not only ambiguous, but actually a genuine bug magnet (it's way too > > easy to accidentally pass a large integer and try to allocate a > > ridiculously large bytes object) > > +1 > > > [...] > > Anyway, what do people think? Does anyone actually *like* the way the > > bytes constructor in Python 3 currently handles integers and want to > > keep it forever? > > Not me! > > > > Does the above proposal sound like a reasonable > > suggestion for improvement in 3.5? Does this hit PEP territory, since > > it's changing the signature and API of a builtin? > > On the vanishingly small chance that there is universal agreement on > this, and a minimum of bike-shedding, I think it would still be useful > to write up a brief half page PEP linking to the discussion here. > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 3 11:39:46 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 02:39:46 -0700 (PDT) Subject: [Python-ideas] loosening the restriction on what types may be unpacked using the ** syntax In-Reply-To: References: Message-ID: <1396517986.88005.YahooMailNeo@web181002.mail.ne1.yahoo.com> From: Eric Snow Sent: Wednesday, April 2, 2014 11:54 PM >T he ** keyword arg unpacking syntax in function calls is useful. > However, currently there is an explicit check for a mapping type > (presumably PyMapping_Check). Actually, I believe there's not really an explicit check. In CPython, ext_do_call?(hg.python.org/cpython/file/default/Python/ceval.c#l4463) uses PyDict_Update (https://docs.python.org/3/c-api/dict.html#c.PyDict_Update) to copy the kwargs argument to its empty dict, and PyDict_Update, unlike the Python equivalent dict.update, "doesn?t fall back to the iterating over a sequence of key value pairs if the second argument has no ?keys? attribute." In PyPy, the relevant code (in?Arguments._combine_starstarargs_wrapped, https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/argument.py#cl-95) is different, but has the same effect (view_as_kwargs only works on mappings). I don't know about other implementations. However, the language does make this an explicit requirement. Section 3.6.4 > So you can pass an object to dict(), > but that same object *may* not work for keyword arg unpacking.? For > example: > >>>> from collections import OrderedDict >>>> def f(**kwargs): return kwargs > ... >>>> kwargs = ((chr(i), i) for i in range(65, 68)) >>>> f(**kwargs) > Traceback (most recent call last): > ? File "", line 1, in > TypeError: f() argument after ** must be a mapping, not generator >>>> f(**OrderedDict(kwargs)) > {'B': 66, 'A': 65, 'C': 67} > > I'd like to relax this restriction so that anything you can pass to > dict is also valid for keyword arg unpacking. As the docs (https://docs.python.org/3.4/library/stdtypes.html#dict) explain, "anything you can pass to dict" is actually one of three things: * 0 or more keyword args * a mapping, followed by 0 or more keyword args * an iterable, followed by 0 or more keyword args I assume you're not suggesting that you should be able to pass keyword args into ** unpacking. So presumably you're just suggesting that ** unpacking should take any iterable, instead of just a mapping, and treat it the same way the dict constructor does. That is: >?If [it] is a mapping object, a dictionary is created with the same key-value pairs as the mapping object. Otherwise, [it] must be an iteratorobject. Each item in the iterable must itself be an iterator with exactly two objects. The first object of each item becomes a key in the new dictionary, and the second object the corresponding value. If a key occurs more than once, the last value for that key becomes the corresponding value in the new dictionary. By the way, I think the example would be a lot less confusing if you didn't use a kwargs parameter.?While we're at it, it would also be less confusing if you used a list instead of a genexpr; as written, if the first call worked, the second call would get an exhausted iterator and therefore have an empty kwargs. And if you just used a dict instead of an OrderedDict, because that's irrelevant to the issue here. So, current behavior: ? ? >>> def f(a, b, c, d): return a, b, c, d ? ? >>> kwargs = [(chr(i), i) for i in range(65, 68)] ? ? >>> f(**dict(kwargs)) ? ? (65, 66, 67, 68) ? ? >>> f(**kwargs) ? ? Traceback (most recent call last): ? ? ? ?File "", line 1, in ? ? ?TypeError: f() argument after ** must be a mapping, not generator Proposed behavior: ? ? >>> def f(a, b, c, d): return a, b, c, d ? ? >>> kwargs = [(chr(i), i) for i in range(65, 68)] ? ? >>> f(**dict(kwargs)) ? ? (65, 66, 67, 68) ? ? >>> f(**kwargs) ? ? (65, 66, 67, 68) This would be a pretty simple change to the language. Section 6.3.4 (https://docs.python.org/3.4/reference/expressions.html#calls) says this: >?If the syntax **expressionappears in the function call, expressionmust evaluate to a mapping, the contents of which are treated as additional keyword arguments. In the case of a keyword appearing in both expressionand as an explicit keyword argument, a TypeErrorexception is raised. Change it to this: > If the syntax **expression appears in the function call, expression must evaluate to an iterable. If the iterable is a mapping, its?contents are treated as additional keyword arguments. Otherwise, each item in the iterable must itself be an iterator with exactly two objects. The first object of each item is treated as an additional keyword, and the second as that keyword's argument.?In the case of a keyword appearing in both expressionand as an explicit keyword argument, a TypeErrorexception is raised. As for implementation, that's also trivial. In CPython, just change ext_do_call to try falling back to PyDict_MergeFromSeq2 if PyDict_Update returns a TypeError. In PyPy, just change view_as_kwargs to work on iterables of iterable pairs. And so on. From abarnert at yahoo.com Thu Apr 3 11:43:30 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 02:43:30 -0700 (PDT) Subject: [Python-ideas] loosening the restriction on what types may be unpacked using the ** syntax In-Reply-To: <1396517986.88005.YahooMailNeo@web181002.mail.ne1.yahoo.com> References: <1396517986.88005.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: <1396518210.11340.YahooMailNeo@web181003.mail.ne1.yahoo.com> Sorry, Yahoo interpreted some key combination as "Send". I think it's mostly readable as-is, but let me clarify a few things. First, if you're proposing what I think you are, I like the idea. I don't know how useful it would be, but I don't see any real downside. So, +0.5. And now, comments inline: From: Andrew Barnert > From: Eric Snow > Sent: Wednesday, April 2, 2014 11:54 PM > >> T he ** keyword arg unpacking syntax in function calls is useful. >> However, currently there is an explicit check for a mapping type >> (presumably PyMapping_Check). > > Actually, I believe there's not really an explicit check. In CPython, > ext_do_call?(hg.python.org/cpython/file/default/Python/ceval.c#l4463) uses > PyDict_Update (https://docs.python.org/3/c-api/dict.html#c.PyDict_Update) to > copy the kwargs argument to its empty dict, and PyDict_Update, unlike the Python > equivalent dict.update, "doesn?t fall back to the iterating over a sequence > of key value pairs if the second argument has no ?keys? attribute." In > PyPy, the relevant code (in?Arguments._combine_starstarargs_wrapped, > https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/argument.py#cl-95) > is different, but has the same effect (view_as_kwargs only works on mappings). I > don't know about other implementations. However, the language reference does make this an explicit requirement.?Section 6.3.4 (https://docs.python.org/3.4/reference/expressions.html#calls) says this: >?If the syntax?**expression?appears in the function call,?expression?must evaluate to a mapping, the contents of which are treated as additional keyword arguments. In the case of a keyword appearing in both?expression?and as an explicit keyword argument, a?TypeError?exception is raised. So basically, you just want to change that paragraph. [snip] ? to: > If the syntax **expression appears in the function call, expression must > evaluate to an iterable. If the iterable is a mapping, its?contents are treated > as additional keyword > arguments. Otherwise, each item in the iterable must itself be an iterator with > exactly two objects. The first object of each item is treated as an additional > keyword, and the second as that keyword's argument.?In the case of a keyword > appearing in both expressionand as an > explicit keyword argument, a TypeErrorexception is raised. > As for implementation, that's also trivial. In CPython, just change? > ext_do_call to try falling back to PyDict_MergeFromSeq2 if PyDict_Update returns > a TypeError. In PyPy, just change view_as_kwargs to work on iterables of > iterable pairs. And so on. From cfkaran2 at gmail.com Thu Apr 3 12:22:37 2014 From: cfkaran2 at gmail.com (Cem Karan) Date: Thu, 3 Apr 2014 06:22:37 -0400 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> <7A6C12B1-A2A5-402F-A4D4-99D523B1E5E7@yahoo.com> <533CC1E0.8030408@canterbury.ac.nz> Message-ID: <5FDAFE36-F670-4DE2-8998-6C1F70B73F34@gmail.com> On Apr 3, 2014, at 12:49 AM, Andrew Barnert wrote: > On Apr 2, 2014, at 19:05, Greg Ewing wrote: > >> On 03/04/14 10:59, Andrew Barnert wrote: >>> [2] In theory these are legal values for an IntRGB16Color, but really, how >>> often do you need literals that are indistinguishable from black but not >>> actually black? >> >> This wouldn't meet the exacting needs of the Assassins' Guild of >> Ankh-Morporkh, who are very fashion-conscious and make quite fine >> distinctions between the shades of black they wear. > > Python 3.x doesn't support Druidic Business Megalithics or any other Big Stone anymore, and with no OutOfCheese error in the stdlib the port to Hex is stalled indefinitely, so you can't seriously be suggesting that this use case matters. I don't see why that would be a serious problem. The standard library still has separate support for both Windows and Unix, so all we need to do is add in another package with support for Hex, and put the OutOfCheese, WantMyFluffyTeddyBear and other errors in there. If you're not running on Hex or other Big Stone, then the package isn't available and you get an ImportError. Problem solved! Thanks, Cem Karan From ron3200 at gmail.com Wed Apr 2 03:45:01 2014 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 01 Apr 2014 21:45:01 -0400 Subject: [Python-ideas] Default value for input In-Reply-To: <20140401115842.GR16526@ando> References: <20140401115842.GR16526@ando> Message-ID: On 04/01/2014 07:58 AM, Steven D'Aprano wrote: > (I'm talking about Python 3 input, previously known as raw_input.) > > I often have the need to include a default or initial value when > asking the user for input. In other words, I'd like to call this: > > input("What is the fish of the day? ", "trout a la creme") > > > and have the prompt be "What is the fish of the day? " and the initial > value in the edit buffer be "trout a la creme", fully editable with > whatever line editing tools are available on the user's system. There > are work-arounds for this lack, but they don't make a nice clean UI. > > Under Linux, with readline, I can implement this relatively simply: One of the things I first looked for when learning python was a get_key_press function. Line input (or whole file input) is great for files, and can make reading data much faster than single character input. But for user input, single char input can be useful. If we could read the keyboard directly with a get_key_press function, then your request along with making curses cross platform, may be fairly easy to do. (The cross platform get_key_press function is the hard part.) Cheers, Ron From random832 at fastmail.us Thu Apr 3 15:28:45 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 03 Apr 2014 09:28:45 -0400 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <533C2AB2.4080905@gmail.com> References: <533C2AB2.4080905@gmail.com> Message-ID: <1396531725.22327.102318737.463E9AAF@webmail.messagingengine.com> On Wed, Apr 2, 2014, at 11:20, Yury Selivanov wrote: > > On 2014-04-02, 12:40 AM, Eric Snow wrote: > > (Inspired by https://mail.python.org/pipermail/python-ideas/2010-October/008532.html) > > > > (color: c, position: p) > While it looks extremely neat, there is one design flaw that is shared > with JavaScript's object literals. ... > So for Python, I think that having unquoted keys in literals is a bad > idea. But namedtuple elements are attributes. This is like objecting to having to do setattr(o,key) instead of o.key when key is in a variable. From random832 at fastmail.us Thu Apr 3 15:32:03 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 03 Apr 2014 09:32:03 -0400 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <20140402212656.GA25514@ando> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> Message-ID: <1396531923.23307.102319381.480C7994@webmail.messagingengine.com> On Wed, Apr 2, 2014, at 17:27, Steven D'Aprano wrote: > But Eric's suggestion skips creating a distinct class and gives you an > instance straight away. So what class will the instance belong to? There > are three obvious possibilities, and they're all unfortunate: > > > (1) every instance is a singleton of a unique class; > > (2) there is a single NamedTuple class that every instance belongs to; > > (3) the namedtuple literal has to cache the keys it has seen, and return > the same class when given the same keys in the same order. (4) Step 3 is done in the compiler, and therefore only namedtuple literals in the same module (or same freshly eval/exec'd string) will have the same type. I think this is what C# does, though you're not supposed to compare them by type, work with their types much, or really store the objects in a non-ephemeral way anyway. Really, it'd almost be nicer to have a _single_ "bag of attributes" type [i.e. class c: pass], and literals for it. From random832 at fastmail.us Thu Apr 3 15:39:49 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 03 Apr 2014 09:39:49 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> On Thu, Mar 20, 2014, at 2:43, Andrew Barnert wrote: > The obvious way to fix this is to make a **kwargs parameter pack the > keyword arguments into an OrderedDict, but Guido has already rejected > that, which is why Eric Snow had to come up with his two other options. > But they don't solve the problem. What about putting a constraint on the dict class that it shall enumerate its keys in the order they were added as long as no key has ever been removed? C#'s Dictionary does this [though it doesn't have a guarantee of doing so]. From p.f.moore at gmail.com Thu Apr 3 15:52:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 3 Apr 2014 14:52:45 +0100 Subject: [Python-ideas] Default value for input In-Reply-To: References: <20140401115842.GR16526@ando> Message-ID: On 2 April 2014 02:45, Ron Adam wrote: > If we could read the keyboard directly with a get_key_press function, then > your request along with making curses cross platform, may be fairly easy to > do. If the aim is cross-platform rich(er) console applications, wouldn't supporting curses on all environments be the easiest answer? AFAIK, there are curses ports for Windows, and it's native to all other platforms Python supports - so just shipping something like pdcurses in the Windows installer would do the trick. Or am I missing some fundamental reason why curses doesn't cover what's needed (and more)? Paul From ericsnowcurrently at gmail.com Thu Apr 3 16:02:26 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Apr 2014 08:02:26 -0600 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: <1396531923.23307.102319381.480C7994@webmail.messagingengine.com> References: <533C2AB2.4080905@gmail.com> <20140402212656.GA25514@ando> <1396531923.23307.102319381.480C7994@webmail.messagingengine.com> Message-ID: On Apr 3, 2014 7:32 AM, wrote: > Really, it'd almost be nicer to have a _single_ "bag of attributes" type > [i.e. class c: pass], and literals for it. So you'd like a literal for types.SimpleNamespace! -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Apr 3 18:41:19 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 03 Apr 2014 09:41:19 -0700 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> Message-ID: <533D8F2F.9040808@stoneleaf.us> On 04/03/2014 06:39 AM, random832 at fastmail.us wrote: > On Thu, Mar 20, 2014, at 2:43, Andrew Barnert wrote: >> The obvious way to fix this is to make a **kwargs parameter pack the >> keyword arguments into an OrderedDict, but Guido has already rejected >> that, which is why Eric Snow had to come up with his two other options. >> But they don't solve the problem. > > What about putting a constraint on the dict class that it shall > enumerate its keys in the order they were added as long as no key has > ever been removed? Wouldn't that destroy the value of random shuffling of keys that was just added? Or was that storage, and this presentation... ? -- ~Ethan~ From random832 at fastmail.us Thu Apr 3 18:55:18 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 03 Apr 2014 12:55:18 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <533D8F2F.9040808@stoneleaf.us> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> <533D8F2F.9040808@stoneleaf.us> Message-ID: <1396544118.25575.102406117.51B30A31@webmail.messagingengine.com> On Thu, Apr 3, 2014, at 12:41, Ethan Furman wrote: > Wouldn't that destroy the value of random shuffling of keys that was just > added? Or was that storage, and this > presentation... ? Can you post a link to information about that? From ethan at stoneleaf.us Thu Apr 3 18:58:41 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 03 Apr 2014 09:58:41 -0700 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396544118.25575.102406117.51B30A31@webmail.messagingengine.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> <533D8F2F.9040808@stoneleaf.us> <1396544118.25575.102406117.51B30A31@webmail.messagingengine.com> Message-ID: <533D9341.3020506@stoneleaf.us> On 04/03/2014 09:55 AM, random832 at fastmail.us wrote: > On Thu, Apr 3, 2014, at 12:41, Ethan Furman wrote: >> Wouldn't that destroy the value of random shuffling of keys that was just >> added? Or was that storage, and this >> presentation... ? > > Can you post a link to information about that? Here's the latest about it in 3.4: https://docs.python.org/3/whatsnew/3.4.html#pep-456-secure-and-interchangeable-hash-algorithm Perhaps always iterating in insertion order would actually make it harder to discover the hash algorithm used? -- ~Ethan~ From storchaka at gmail.com Thu Apr 3 21:03:19 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 03 Apr 2014 22:03:19 +0300 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> Message-ID: 02.04.14 14:40, Nick Coghlan ???????(??): > I thought of that, but it seems like a recipe for typos and confusion. > bytes.byte and bytearray.byte seem clearer and safer. bytearray.byte looks deceptive. It returns not a byte, but 1-element bytearray. I doubt that creating 1-element bytearray is enough often case to add new special method (unlike to bytes.byte). From random832 at fastmail.us Thu Apr 3 21:14:27 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 03 Apr 2014 15:14:27 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <533D9341.3020506@stoneleaf.us> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> <533D8F2F.9040808@stoneleaf.us> <1396544118.25575.102406117.51B30A31@webmail.messagingengine.com> <533D9341.3020506@stoneleaf.us> Message-ID: <1396552467.4459.102458513.46EBCB1F@webmail.messagingengine.com> On Thu, Apr 3, 2014, at 12:58, Ethan Furman wrote: > On 04/03/2014 09:55 AM, random832 at fastmail.us wrote: > > On Thu, Apr 3, 2014, at 12:41, Ethan Furman wrote: > >> Wouldn't that destroy the value of random shuffling of keys that was just > >> added? Or was that storage, and this > >> presentation... ? > > > > Can you post a link to information about that? > > Here's the latest about it in 3.4: > > https://docs.python.org/3/whatsnew/3.4.html#pep-456-secure-and-interchangeable-hash-algorithm > > Perhaps always iterating in insertion order would actually make it harder > to discover the hash algorithm used? Iterating in insertion order would make the iteration completely independent of the hashing algorithm. The hashing algorithm determines what buckets each item goes in, but you don't _have_ to iterate in bucket order, depending on your data structure. C#, for example, stores items in an array [which are populated from a freelist of array slots that don't have items in them, or sequentially at the end of the array if there are no slots in the freelist] and the buckets as linked lists within the array. The security purpose of that change is to prevent an attacker from making a bunch of items all go in the same bucket, ruining the performance characteristics of the hash table. From ncoghlan at gmail.com Thu Apr 3 22:04:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 4 Apr 2014 06:04:07 +1000 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> Message-ID: On 4 Apr 2014 05:03, "Serhiy Storchaka" wrote: > > 02.04.14 14:40, Nick Coghlan ???????(??): > >> I thought of that, but it seems like a recipe for typos and confusion. >> bytes.byte and bytearray.byte seem clearer and safer. > > > bytearray.byte looks deceptive. It returns not a byte, but 1-element bytearray. > > I doubt that creating 1-element bytearray is enough often case to add new special method (unlike to bytes.byte). I actually agree, but Guido preferred the greater API consistency. Since I'm only -0 on bytearray.byte, I don't have much motivation to argue about it. Cheers, Nick. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Apr 3 22:17:54 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Apr 2014 13:17:54 -0700 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> Message-ID: Actually, I hadsn't thought about that much, it's fine to only have bytes.byte(). On Thu, Apr 3, 2014 at 1:04 PM, Nick Coghlan wrote: > > On 4 Apr 2014 05:03, "Serhiy Storchaka" wrote: > > > > 02.04.14 14:40, Nick Coghlan ???????(??): > > > >> I thought of that, but it seems like a recipe for typos and confusion. > >> bytes.byte and bytearray.byte seem clearer and safer. > > > > > > bytearray.byte looks deceptive. It returns not a byte, but 1-element > bytearray. > > > > I doubt that creating 1-element bytearray is enough often case to add > new special method (unlike to bytes.byte). > > I actually agree, but Guido preferred the greater API consistency. Since > I'm only -0 on bytearray.byte, I don't have much motivation to argue about > it. > > Cheers, > Nick. > > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Apr 3 23:49:58 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 4 Apr 2014 07:49:58 +1000 Subject: [Python-ideas] Fixing the Python 3 bytes constructor In-Reply-To: References: <5336F9A6.2030006@stoneleaf.us> <533B75F5.9040601@stoneleaf.us> Message-ID: On 4 Apr 2014 06:18, "Guido van Rossum" wrote: > > Actually, I hadsn't thought about that much, it's fine to only have bytes.byte(). Oh, cool, I guess I *should* have argued about that one :) Cheers, Nick. > > > On Thu, Apr 3, 2014 at 1:04 PM, Nick Coghlan wrote: >> >> >> On 4 Apr 2014 05:03, "Serhiy Storchaka" wrote: >> > >> > 02.04.14 14:40, Nick Coghlan ???????(??): >> > >> >> I thought of that, but it seems like a recipe for typos and confusion. >> >> bytes.byte and bytearray.byte seem clearer and safer. >> > >> > >> > bytearray.byte looks deceptive. It returns not a byte, but 1-element bytearray. >> > >> > I doubt that creating 1-element bytearray is enough often case to add new special method (unlike to bytes.byte). >> >> I actually agree, but Guido preferred the greater API consistency. Since I'm only -0 on bytearray.byte, I don't have much motivation to argue about it. >> >> Cheers, >> Nick. >> >> > >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Apr 4 00:24:00 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 4 Apr 2014 09:24:00 +1100 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396552467.4459.102458513.46EBCB1F@webmail.messagingengine.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> <533D8F2F.9040808@stoneleaf.us> <1396544118.25575.102406117.51B30A31@webmail.messagingengine.com> <533D9341.3020506@stoneleaf.us> <1396552467.4459.102458513.46EBCB1F@webmail.messagingengine.com> Message-ID: On Fri, Apr 4, 2014 at 6:14 AM, wrote: > Iterating in insertion order would make the iteration completely > independent of the hashing algorithm. The hashing algorithm determines > what buckets each item goes in, but you don't _have_ to iterate in > bucket order, depending on your data structure. C#, for example, stores > items in an array [which are populated from a freelist of array slots > that don't have items in them, or sequentially at the end of the array > if there are no slots in the freelist] and the buckets as linked lists > within the array. However that's done, it implies additional storage beyond the straight-forward mapping, right? Python's dict implementation is *highly* optimized (which is necessary, given how much of Python is built on dicts), and any proposal that basically says "Every dict needs to keep track of insertion order, for the sake of the handful of cases where you actually care" is likely to have the aerodynamic qualities of a rock. ChrisA From ericsnowcurrently at gmail.com Fri Apr 4 01:37:03 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Apr 2014 17:37:03 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: On Apr 3, 2014 2:55 AM, "Andrew Barnert" wrote: > It's definitely a big deal. In Python 3.4, it is trivial to write a wrapper that perfectly forwards its arguments. Or that modifies its arguments in somewhat, then perfectly forwards the modified version. This is critical to being able to write decorators. Nearly every existing decorator in the stdlib, third-party libs, ActiveState recipes, and even your own projects does this. I'll show a dead-simple example below. > > > But with either of your options, that would no longer be true. It would be at least a little more difficult to write a wrapper that perfectly forwards its arguments, and every existing decorator in the world would have to be rewritten to do so. I'll concede "a little more difficult" in the pass-through case (with unaware decorators being the main issue). :) One reason why I don't think it's a huge problem in the decorator case is that you decorate your own functions (though not necessarily with your own decorators). So if you change an existing function to use ordered kwargs, then you would already be focused on the feature. You would ensure any decorators you've applied to the function accommodate ordered pass-through, perhaps adapting/wrapping the decorators to do so. It's all pretty localized in your code, so it should be hard to miss. I agree that both of the approaches I've mentioned require this extra work. I also agree that it would be good to avoid this complication. I have a few more ideas which I'll include in the upcoming PEP (and I've summarized below). Other than the pass-through case, the 2 outstanding approaches seem fine. Existing functions would probably need a signature change anyway and for new functions no one has to change any code. :) Either way, if a function depends on ordered kwargs, then that's the sort of important feature that you will make clear to people using your function, on par with the rest of the signature. [snipped example] > Can Python fix this for you magically? I'll drop this proposal in a heartbeat if it requires something like that! I just don't think it does, even with the 2 less-optimal solutions we've been discussing. (My definition of magic may be different than yours though. ) > > The only way to make eggs end up as order-preserving is to change Python so that all functions (or at least all functions defined inside other functions) are order-preserving. > > Either you or someone else suggested that if Python could just pass the kwargs dict straight through instead of unpacking and repacking it, that would help. But it wouldn't. Agreed. (Pretty sure that wasn't me. :) ) That would be a non-option for other reasons anyway. [more snipped] > Or consider your original example (with explicit a and b params on spam before **kwargs) with memoize, where the kwargs in eggs has a, b, c, and d, but the one inside spam has only c and d. That should be a non-issue given how the data is packed into a new kwargs. > > Now, obviously you could fix any particular wrapper. Presumably your magic decorator works by doing something detectable from Python, like a new co_flags flag. So, we could define two new functions (presumably in inspect and functools, respectively): > > > def preserves_kwargs_order(func): > return func.__code__.co_flags & 16 > > def wrap_preserve_kwargs_order(func): > def wrapper(inner): > if inspect.preserves_kwargs_order(func): > return preserve_kwargs_order(inner) > else: > return inner > return wrapper > > And now, all you have to do is add one line to most decorators and they're once again perfect forwarders: > > def memoize(func): > > _cache = {} > @functools.wrap_preserve_kwargs_order(func) > def wrapper(*args, **kwargs): > if (args, kwargs) not in _cache: > _cache[args, kwargs] = func(*args, **kwargs) > return _cache[args, kwargs] > return wrapper > > > But the point is, you still have to add that decorator to every wrapper function in the world or they're no longer perfect forwarders. Nice. > > You could even add that wrap_preserve_kwargs_order call into functools.wraps, and that would fix _many_ decorators (because many decorators are written to use functools.wraps), but still not all. Also nice. That said, I agree that it would be best to come up with a solution that drops in without breaking the "perfect forwarders", thus rendering those tools unnecessary. Here are 3 ideas I had last night and today: 1. Add a '__order__' attribute to dict that defaults to None, and always set it for kwargs (to a frozen list/tuple). 2. Use a minimal dict subclass just for kwargs that provides __order__ (meaning all other uses of dict don't take the memory hit). 3. Use OrderedDict for kwargs by default, and provide a decorator that people can use to get just a dict (a.k.a. the current behavior). It's only a few extreme cases where OrderedDict is problematic. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Apr 4 01:41:40 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Apr 2014 17:41:40 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396532389.25136.102321649.227B069F@webmail.messagingengine.com> <533D8F2F.9040808@stoneleaf.us> <1396544118.25575.102406117.51B30A31@webmail.messagingengine.com> <533D9341.3020506@stoneleaf.us> <1396552467.4459.102458513.46EBCB1F@webmail.messagingengine.com> Message-ID: On Thu, Apr 3, 2014 at 4:24 PM, Chris Angelico wrote: > On Fri, Apr 4, 2014 at 6:14 AM, wrote: > > Iterating in insertion order would make the iteration completely > > independent of the hashing algorithm. The hashing algorithm determines > > what buckets each item goes in, but you don't _have_ to iterate in > > bucket order, depending on your data structure. C#, for example, stores > > items in an array [which are populated from a freelist of array slots > > that don't have items in them, or sequentially at the end of the array > > if there are no slots in the freelist] and the buckets as linked lists > > within the array. > > However that's done, it implies additional storage beyond the > straight-forward mapping, right? Python's dict implementation is > *highly* optimized (which is necessary, given how much of Python is > built on dicts), and any proposal that basically says "Every dict > needs to keep track of insertion order, for the sake of the handful of > cases where you actually care" is likely to have the aerodynamic > qualities of a rock. > Raymond has a proposal (back-burnered?) for a compact dict implementation that preserves order in iteration until the first deletion: https://mail.python.org/pipermail/python-dev/2012-December/123028.html https://mail.python.org/pipermail/python-dev/2013-May/126327.html -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Apr 4 02:41:21 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 17:41:21 -0700 (PDT) Subject: [Python-ideas] Default value for input In-Reply-To: References: <20140401115842.GR16526@ando> Message-ID: <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> From: Ron Adam Sent: Tuesday, April 1, 2014 6:45 PM > On 04/01/2014 07:58 AM, Steven D'Aprano wrote: >> I often have the need to include a default or initial value when >> asking the user for input. In other words, I'd like to call this: >> >> input("What is the fish of the day? ", "trout a la > creme") >> >> >> and have the prompt be "What is the fish of the day? " and the > initial >> value in the edit buffer be "trout a la creme", fully editable > with >> whatever line editing tools are available on the user's system. There >> are work-arounds for this lack, but they don't make a nice clean UI. >> >> Under Linux, with readline, I can implement this relatively simply: > One of the things I first looked for when learning python was a > get_key_press function. > > Line input (or whole file input) is great for files, and can make reading > data much faster than single character input.? But for user input, single > char input can be useful. > > If we could read the keyboard directly with a get_key_press function,? then > your request along with making curses cross platform, may be fairly easy to do. >? > (The cross platform get_key_press function is the hard part.) Are you serious? A get_key_press function is the _easy_ part. On Unix, you use termios to put stdin into raw mode then select or blocking-read stdin; on Windows, you use the msvcrt module to call the wrappers around the console I/O functions. But what do you think those functions return for, say, left arrow, or ctrl+end? (Even beyond non-character keys, even the characters don't necessarily show up as a single character, but as 1 or more bytes in the appropriate encoding (Unix) or 1 or more UTF-16 codes (Windows), so your state is less trivial than you're expecting?) And what are you going to do with a left-arrow key press when you get it??If your answer is "move the cursor left", how do you do that? On Windows you use console I/O functions; on Unix, you talk to termcap to figure out which control sequence to send and then send that; etc. And, once you've moved the cursor left, you probably want to insert rather than overwriting.?And of course you need to be able to deal with input that wraps onto multiple lines; you may or may not have to do that wrapping yourself. And so on. And once you've solved all of that, and figured out how to keep track of buffers and cursor positions, then you just need to write code to emulate everything that each platform does for terminal line editing. To get an idea of how much work that is for each platform, look at the source to readline or libedit. This is exactly why Python doesn't do any of that; it just uses readline (or libedit's readline-compat mode) on Unix, lets cmd.exe do its thing on Windows, etc. (This also makes it relatively easy for IDEs to hook the behavior and, e.g., make input give you a GUI window with a Tk text entry box.) From abarnert at yahoo.com Fri Apr 4 02:55:14 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 17:55:14 -0700 Subject: [Python-ideas] Default value for input In-Reply-To: References: <20140401115842.GR16526@ando> Message-ID: On Apr 3, 2014, at 6:52, Paul Moore wrote: > If the aim is cross-platform rich(er) console applications, wouldn't > supporting curses on all environments be the easiest answer? AFAIK, > there are curses ports for Windows, and it's native to all other > platforms Python supports - so just shipping something like pdcurses > in the Windows installer would do the trick. Or am I missing some > fundamental reason why curses doesn't cover what's needed (and more)? First, is pdcurses still a live/working project? Does it work with Python's curses module? If so, that might be worth doing entirely independent of whether it covers what's needed for this thread. I suspect the main argument against it is that there are so few people writing curses apps in Python who care about Windows that nobody's bothered to even ask for it, much less do it. Meanwhile, if you go back to the start of this thread, really he wants normal terminal I/O with readline-like input or the platform's normal equivalent, except that he can pre-fill the input buffer. I think that would be a lot more work with curses than with readline. The question is whether there's a way to do it cross-platform, e.g., with cmd.exe line-editing. If so, that's probably easier than using curses. If not, then curses is certainly better than there being no way to do it, or than emulating line editing from scratch on top of console I/O and/or control sequences. From abarnert at yahoo.com Fri Apr 4 03:50:19 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 18:50:19 -0700 (PDT) Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> From: Eric Snow Sent: Thursday, April 3, 2014 4:37 PM >On Apr 3, 2014 2:55 AM, "Andrew Barnert" wrote: >> It's definitely a big deal. In Python 3.4, it is trivial to write a wrapper that perfectly forwards its arguments. Or that modifies its arguments in somewhat, then perfectly forwards the modified version. This is critical to being able to write decorators. Nearly every existing decorator in the stdlib, third-party libs, ActiveState recipes, and even your own projects does this. I'll show a dead-simple example below. >> >> But with either of your options, that would no longer be true. It would be at least a little more difficult to write a wrapper that perfectly forwards its arguments, and every existing decorator in the world would have to be rewritten to do so. > >I'll concede "a little more difficult" in the pass-through case (with unaware decorators being the main issue). ?:) ?One reason why I don't think it's a huge problem in the decorator case is that you decorate your own functions (though not necessarily with your own decorators). ?So if you change an existing function to use ordered kwargs, then you would already be focused on the feature. ?You would ensure any decorators you've applied to the function accommodate ordered pass-through, perhaps adapting/wrapping the decorators to do so. ?It's all pretty localized in your code, so it should be hard to miss. First, not all wrappers are usually used as decorators. The most obvious example is partial, which is why I raised it before. I've also applied lru_cache, or a TTL-based cache that I got from some module on PyPI, to lookup functions that I pulled out of another module. And so on. Any such general-purpose wrappers that work today would need to be replaced after your change.?It's not at all localized in my code; it's not even all in my code.?The guy who wrote TTLCache has no idea whether I want to use it for keyword-order-preserving functions. (Well, today, he knows that I _don't_ want to do so, because it's not possible?)? >Other than the pass-through case, the 2 outstanding approaches seem fine. ?Existing functions would probably need a signature change anyway and for new functions no one has to change any code. :) ?Either way, if a function depends on ordered kwargs, then that's the sort of important feature that you will make clear to people using your function, on par with the rest of the signature. Well, that has nothing to do with the point I raised, but actually, it raises another (smaller) problem. The fact that a function has been wrapped in a decorator doesn't change its signature, as in the thing you can inspect with things like functools.getfullargspec or functools.signature. So, just adding this decorator, or even adding an inspect function that looks for whatever the decorator does (like setting a new bit on co_flags), isn't sufficient; you need to work out how it should affect inspect.Signature and/or inspect.Parameter objects. (Maybe there's just a new kind, VAR_ORDERED_KEYWORD? but the fact that Parameters of this kind would have to have the same str() as VAR_KEYWORD means most people still wouldn't see it?) >> Can Python fix this for you magically? > >I'll drop this proposal in a heartbeat if it requires something like that! ?I just don't think it does, even with the 2 less-optimal solutions we've been discussing. ?(My definition of magic may be different than yours though. ) Well, you said before that the perfect-forwarding wrapper problem could be solved easily. I asked you to explain how. Now it seems like your answer is that it can't be solved, short of by rewriting every wrapper function that might ever be used on order-preserving functions. Which is exactly what I said, and you denied, in the first place. >> But the point is, you still have to add that decorator to every wrapper function in the world or they're no longer perfect forwarders. > >Nice. The decorator may be nice, but the fact that you have to add that decorator to ever wrapper function defined in every general-purpose decorator or other wrapper-generator in the world does not seem nice to me. >That said, I agree that it would be best to come up with a solution that drops in without breaking the "perfect forwarders", thus rendering those tools unnecessary. ?Here are 3 ideas I had last night and today: > > >1. Add a '__order__' attribute to dict that defaults to None, and always set it for kwargs (to a frozen list/tuple). A dict with an __order__ attrib is just a less-useful but nearly-as-heavy-weight equivalent to an OrderedDict. What about the cost of always using an OrderedDict makes it unacceptable? How much better is this? And it doesn't actually work. First, the **d calling syntax unpacks the keywords in d's iteration order. That works fine for OrderedDict, but not for a dict that has an order but ignores it while iterating. So you need an additional rule (and additional code in each implementation) to make the **d calling syntax use __order__ order instead of iteration order (and deal with all of the edge cases, like when __order__ is missing some keys, or has keys the dict doesn't, etc.). And, even after that, any wrapper that modifies its kwargs instead of passing them through unchanged (like, again, partial) could easily break __order__ unless you also change all such wrappers to handle it properly. Again, not a problem for OrderedDict, because OrderedDict automatically handles it properly. >2. Use a minimal dict subclass just for kwargs that provides __order__ (meaning all other uses of dict don't take the memory hit). That still has the other problems as #1. >3. Use OrderedDict for kwargs by default, and provide a decorator that people can use to get just a dict (a.k.a. the current behavior). ?It's only a few extreme cases where OrderedDict is problematic. So how do C functions (which I'm guessing are many of those extreme cases?) get just a dict? Also, is it actually true that Guido rejected the idea of "kwargs is always an OrderedDict" out of hand because a few extreme cases are problematic, or is that just a guess? From ericsnowcurrently at gmail.com Fri Apr 4 05:03:56 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 3 Apr 2014 21:03:56 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: On Thu, Apr 3, 2014 at 7:50 PM, Andrew Barnert wrote: > First, not all wrappers are usually used as decorators. The most obvious example is partial, which is why I raised it before. I've also applied lru_cache, or a TTL-based cache that I got from some module on PyPI, to lookup functions that I pulled out of another module. And so on. Any such general-purpose wrappers that work today would need to be replaced after your change. It's not at all localized in my code; it's not even all in my code. The guy who wrote TTLCache has no idea whether I want to use it for keyword-order-preserving functions. (Well, today, he knows that I _don't_ want to do so, because it's not possible...) So you are saying that it is not uncommon to use "perfect passthrough" decorators/wrappers you do not control. In my experience it is rare outside the stdlib, but then again I gladly admit that my experience has not involved software domains where apparently such takes place. Regardless, the workarounds are the same. Likewise they add the same complication. Perhaps it does favor the decorator syntax over the __kworder__ local variable. > Well, that has nothing to do with the point I raised, but actually, it raises another (smaller) problem. The fact that a function has been wrapped in a decorator doesn't change its signature, as in the thing you can inspect with things like functools.getfullargspec or functools.signature. > > So, just adding this decorator, or even adding an inspect function that looks for whatever the decorator does (like setting a new bit on co_flags), isn't sufficient; you need to work out how it should affect inspect.Signature and/or inspect.Parameter objects. (Maybe there's just a new kind, VAR_ORDERED_KEYWORD... but the fact that Parameters of this kind would have to have the same str() as VAR_KEYWORD means most people still wouldn't see it...) Looking into the implications on inspect.signature, et al. is on my to-do list. :) > Well, you said before that the perfect-forwarding wrapper problem could be solved easily. I asked you to explain how. Now it seems like your answer is that it can't be solved, short of by rewriting every wrapper function that might ever be used on order-preserving functions. Which is exactly what I said, and you denied, in the first place. I stand by what I said. I don't think the impact is that severe and I think that solving the issue wouldn't be that hard and I do agree that it adds complication that would be worth avoiding. I think your summary exaggerates things a bit. You are correct that I did not give any concrete solution to resolve the perfect-forwarding wrapper case, even in the face of your concrete example. If, when we have a better idea of the various options, one of the original two (or any other opt-in solution) stands above the rest I'll be glad to spend more time looking on how to make the pass-through case work with less complication. > >>> But the point is, you still have to add that decorator to every wrapper function in the world or they're no longer perfect forwarders. >> >>Nice. > > > The decorator may be nice, but the fact that you have to add that decorator to ever wrapper function defined in every general-purpose decorator or other wrapper-generator in the world does not seem nice to me. It only matters for the ones you are applying to your functions that need to preserve keyword order. I agree that any fix that you can't easily apply to a function you do not "own" isn't very helpful. > >>That said, I agree that it would be best to come up with a solution that drops in without breaking the "perfect forwarders", thus rendering those tools unnecessary. Here are 3 ideas I had last night and today: > >> >> >>1. Add a '__order__' attribute to dict that defaults to None, and always set it for kwargs (to a frozen list/tuple). > > A dict with an __order__ attrib is just a less-useful but nearly-as-heavy-weight equivalent to an OrderedDict. What about the cost of always using an OrderedDict makes it unacceptable? How much better is this? Guido's concern was that in some uncommon cases the difference in performance (both memory and speed) would render OrderedDict undesirable for **kwargs. By storing the initial order on a dict attribute the only overhead would be 1 pointer per dict plus the size of the container. > > And it doesn't actually work. > > First, the **d calling syntax unpacks the keywords in d's iteration order. That works fine for OrderedDict, but not for a dict that has an order but ignores it while iterating. So you need an additional rule (and additional code in each implementation) to make the **d calling syntax use __order__ order instead of iteration order (and deal with all of the edge cases, like when __order__ is missing some keys, or has keys the dict doesn't, etc.). > > And, even after that, any wrapper that modifies its kwargs instead of passing them through unchanged (like, again, partial) could easily break __order__ unless you also change all such wrappers to handle it properly. Again, not a problem for OrderedDict, because OrderedDict automatically handles it properly. Correct, the interpreter would have to take __order__ into account when unpacking kwargs into a function call. I realized earlier that I hadn't mentioned that. > >>2. Use a minimal dict subclass just for kwargs that provides __order__ (meaning all other uses of dict don't take the memory hit). > > > That still has the other problems as #1. > >>3. Use OrderedDict for kwargs by default, and provide a decorator that people can use to get just a dict (a.k.a. the current behavior). It's only a few extreme cases where OrderedDict is problematic. > > > So how do C functions (which I'm guessing are many of those extreme cases...) get just a dict? Good question. I already plan on addressing the impact of the proposal on the C-API as part of the PEP. > > Also, is it actually true that Guido rejected the idea of "kwargs is always an OrderedDict" out of hand because a few extreme cases are problematic, or is that just a guess? > He indicated that any solution would have to demonstrate that it did not have any significant impact on performance in those edge cases. [1][2][3] -eric [1] https://mail.python.org/pipermail/python-dev/2013-May/126328.html [2] https://mail.python.org/pipermail/python-ideas/2013-February/019699.html [3] https://mail.python.org/pipermail/python-ideas/2013-February/019704.html From ron3200 at gmail.com Fri Apr 4 06:25:12 2014 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 04 Apr 2014 00:25:12 -0400 Subject: [Python-ideas] Default value for input In-Reply-To: <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <20140401115842.GR16526@ando> <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: On 04/03/2014 08:41 PM, Andrew Barnert wrote: > From: Ron Adam > > Sent: Tuesday, April 1, 2014 6:45 PM > > >> >On 04/01/2014 07:58 AM, Steven D'Aprano wrote: > >>> >> I often have the need to include a default or initial value when >>> >> asking the user for input. In other words, I'd like to call this: >>> >> >>> >> input("What is the fish of the day? ", "trout a la >> >creme") >>> >> >>> >> >>> >> and have the prompt be "What is the fish of the day? " and the >> >initial >>> >> value in the edit buffer be "trout a la creme", fully editable >> >with >>> >> whatever line editing tools are available on the user's system. There >>> >> are work-arounds for this lack, but they don't make a nice clean UI. >>> >> >>> >> Under Linux, with readline, I can implement this relatively simply: > >> >One of the things I first looked for when learning python was a >> >get_key_press function. >> > >> >Line input (or whole file input) is great for files, and can make reading >> >data much faster than single character input. But for user input, single >> >char input can be useful. >> > >> >If we could read the keyboard directly with a get_key_press function, then >> >your request along with making curses cross platform, may be fairly easy to do. >> > >> >(The cross platform get_key_press function is the hard part.) > > Are you serious? Sure it's only a starting point, not a end solution by it self. I was including the parts you you mention below, taking into account the platform and console differences and returning a Unicode string. I wasn't thinking of simply reading and returning the raw key codes. Just get the input (including escape sequences) with out echoing it to the screen. Probably the most straight forward way would be to import a console class, and initiate an instance, then use methods on it to read the input and write the output. That way all the platform stuff can be abstracted away. A higher level function that uses that class, would handle updating the line buffer and write those updates out to the screen. (by calling methods on the instance. > A get_key_press function is the_easy_ part. On Unix, you use termios to > put stdin into raw mode then select or blocking-read stdin; on Windows, > you use the msvcrt module to call the wrappers around the console I/O > functions. > But what do you think those functions return for, say, left arrow, or > ctrl+end? (Even beyond non-character keys, even the characters don't > necessarily show up as a single character, but as 1 or more bytes in the > appropriate encoding (Unix) or 1 or more UTF-16 codes (Windows), so your > state is less trivial than you're expecting?) Yes, there are a lot of small details to work out. But I don't think any of those are impossible to manage. > And what are you going to do with a left-arrow key press when you get > it? If your answer is "move the cursor left", how do you do that? On > Windows you use console I/O functions; on Unix, you talk to termcap to > figure out which control sequence to send and then send that; etc. > > And, once you've moved the cursor left, you probably want to insert > rather than overwriting. And of course you need to be able to deal with > input that wraps onto multiple lines; you may or may not have to do that > wrapping yourself. And so on. > > And once you've solved all of that, and figured out how to keep track of > buffers and cursor positions, then you just need to write code to > emulate everything that each platform does for terminal line editing. To > get an idea of how much work that is for each platform, look at the > source to readline or libedit. I agree although it's been a while sense I looked at the specific details. > This is exactly why Python doesn't do any of that; it just uses readline > (or libedit's readline-compat mode) on Unix, lets cmd.exe do its thing > on Windows, etc. (This also makes it relatively easy for IDEs to hook > the behavior and, e.g., make input give you a GUI window with a Tk text > entry box.) Those will still work. I don't think anyone is suggesting replacing all of pythons console related code. Cheers, Ron From abarnert at yahoo.com Fri Apr 4 07:42:41 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 22:42:41 -0700 (PDT) Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <1396590161.64397.YahooMailNeo@web181003.mail.ne1.yahoo.com> From: Eric Snow Sent: Thursday, April 3, 2014 8:03 PM > On Thu, Apr 3, 2014 at 7:50 PM, Andrew Barnert wrote: [snip] > So you are saying that it is not uncommon to use "perfect passthrough" > decorators/wrappers you do not control. Yes. Just my first example, partial, is ridiculously common, and it's a wrapper I don't control.?Half the servers, GUIs, and other callback-based programs I've worked on need partial functions, and get them either by using partial, or by using?lambda functions instead?which has an equivalent and parallel problem that, come to think of it, still requires a solution too? However, what we _don't_ know is how common it would be to use such wrappers with functions that need ordered kwargs. The prototypical case of using partial on GUI or server callbacks?obviously doesn't apply there?in a Tkinter GUI, your callback functions almost never take kwargs, and, if they did, it would probably be just to pass the dict as a widget configure dict or something like that, which couldn't possibly care about the order. But really, _no_ example that works today could apply; it's not possible to write functions that care about their keyword order, so we can only try to guess whether the potential use cases for them could involve wrapping them or not. > Regardless, the workarounds are the same.? Likewise they add the same > complication.? Perhaps it does favor the decorator syntax over the > __kworder__ local variable. I'm not sure it favors either. Either way, every function that returns a forwarding wrapper will need to be changed to apply the workaround. The workaround itself?may be easier, or at least prettier, with the decorator, but the hard part isn't the workaround itself (as I showed, it could just be a simple on-line change), but figuring out which wrappers need it, and possibly copying and forking them into your own code, so you can apply the workaround to each one. [snip] > Looking into the implications on inspect.signature, et al. is on my > to-do list. :) After a little more thought, what's really needed is a consistent way to represent ordered-kwargs parameters in *documentation*. Once you have that, whatever it is, inspect.signature should be simple. For example, if you use kind=VAR_ORDERED_KEYWORD, then it's just a matter of making Parameter.__str__ return whatever-it-is that the docs would have instead of returning the normal **{name}. This does give a little more weight to proposals that do something visible in the signature itself?the ***okwargs, or an annotation, or whatever. But I think all of them have enough down-sides compared to your more favored versions that it wouldn't be nearly enough weight to shift things. [snip] > It only matters for the ones you are applying to your functions that > need to preserve keyword order.? I agree that any fix that you can't > easily apply to a function you do not "own" isn't very helpful. Yes, that last part is basically my main point. >>> 1. Add a '__order__' attribute to dict that defaults to None, > and always set it for kwargs (to a frozen list/tuple). >> >> A dict with an __order__ attrib is just a less-useful but > nearly-as-heavy-weight equivalent to an OrderedDict. What about the cost of > always using an OrderedDict makes it unacceptable? How much better is this? > > Guido's concern was that in some uncommon cases the difference in > performance (both memory and speed) would render OrderedDict > undesirable for? **kwargs.? By storing the initial order on a dict > attribute the only overhead would be 1 pointer per dict plus the size > of the container. First, after reading the messages you linked, I don't think your characterizing Guido's concerns fairly. His first message is not about uncommon cases at all, it's about the fact that every function call (at least every one that uses kwargs) will slow down noticeably: "My?main concern is speed -- since?most code doesn't need it and function calls are already slow (and?obviously very common :-) it would be a shame if this slowed down?function calls that don't need it noticeably." It's hard to predict how much of a slowdown a nice C OrderedDict would cause here, and how much less of a slowdown just creating and using __order__ would cause. But from a quick simulation in pure Python, it looks like you're just turning a 2.7x slowdown into a 2.2x slowdown?better, but still clearly noticeable, and therefore not acceptable. This means you need some way to make sure functions that don't ask for ordered kwargs don't get it (unless you can actually make it not noticeably slower). I think all of your solutions (including storing __kworder__) will be noticeably slower; there's just no way around the fact that maintaining a dict and its order and iterating it in custom order takes longer than maintaining a dict and iterating it in natural order.?Trying to disguise the ordering info, or make it not quite as slow, or make it opt-out instead of opt-in, etc., doesn't solve this problem. Only?finding a way to do nothing at all when nobody wants it solves this problem. At which point you might as well do the simplest thing?use OrderedDict?for cases where people _do_ want it. And that comes back to my original point: Where does someone want order? When they explicitly ask for it, but also when wrapping a function that asks for it in a generic wrapper. And that's the hard part; inside the wrapper (in some library code), you don't know to ask for it; outside the wrapper (in your app), there's no way to do it. Meanwhile, the "uncommon" cases (which you elsewhere call "rare", "extreme", and "edge cases") are a different story. I don't think Guido thinks they're very uncommon. He says, "I write code regularly that?" saves kwargs for later, and possibly even piles other stuff into it. His primary issue here is that having anything other than a plain dict is wrong and confusing. Secondarily, he also mentions "possible" performance problems. Here is where the memory comes in. Increasing the memory of every kwargs dict by 25% instead of 75% is nice, but I don't think it solves the memory problem, and I don't think the memory problem is the main issue anyway. This seems to rule out any solution that gives you a modified kwargs dict when you don't ask for it, whether it's an OrderedDict or a dict with an __order__ attribute or anything else.?I now understand why you proposed the separate __kworder__ magic parameter solution. But really, given the first problem, any solution that adds _any_ noticeable overhead when you don't ask for it is already ruled out, so trying to put the extra overhead somewhere other than the dict itself doesn't help. You need to actually not create it. Anyway, after a little more back-burner thought, I actually did think of some magic that might work. But?I don't think you, or anyone else, will like it. I certainly don't. At compile time, any code object that has a call to something accessed via a closure cell gets a calls-local flag. (This should catch all decorators and other wrapper-generators, but many false positives as well.) At runtime, when building a function from a code object with the calls-local flag, if any of the __closure__ members are functions with the order-kwargs flag, the new function also gets the flag as well. (This eliminates most of the false positives, without eliminating any actual wrappers unless they're designed pathologically. And if they are pathological, well, then they lose kwargs ordering.)?That's all only a couple lines of code in the compiler and interpreter.?Tada? Not sure; the idea is too horrible and non-Pythonic to think through all the way. Ultimately, this design would mean that all local functions get ordered kwargs, but there's an optimization that usually eliminates the order when you don't need it? which is just wrong. From abarnert at yahoo.com Fri Apr 4 08:01:32 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 3 Apr 2014 23:01:32 -0700 (PDT) Subject: [Python-ideas] Default value for input In-Reply-To: References: <20140401115842.GR16526@ando> <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <1396591292.48940.YahooMailNeo@web181003.mail.ne1.yahoo.com> From: Ron Adam Sent: Thursday, April 3, 2014 9:25 PM > On 04/03/2014 08:41 PM, Andrew Barnert wrote: >> From: Ron Adam >>> If we could read the keyboard directly with a get_key_press >>>?function,? then >>> your request along with making curses cross platform, may be fairly >>> easy to do. >>> (The cross platform get_key_press function is the hard part.) >> >> Are you serious? > > Sure it's only a starting point, not a end solution by it self.? I was > including the parts you you mention below, taking into account the platform > and console differences and returning a Unicode string.? I wasn't thinking > of simply reading and returning the raw key codes.? Just get the input > (including escape sequences) with out echoing it to the screen. Again, that's easy to do on both platforms, but it doesn't even begin to approach making the complete solution easy. > Probably the most straight forward way would be to import a console class, > and initiate an instance, then use methods on it to read the input and > write the output.? That way all the platform stuff can be abstracted away. > > A higher level function that uses that class, would handle updating the > line buffer and write those updates out to the screen.? (by calling methods > on the instance. There are only two things you could mean here. 1. The console class is basically a wrapper around readline or its equivalent, in which case we have no need for get_key_press or any of the other stuff you asked about. 2. The console class is just the low-level stuff that's implementable on all platforms, like reading and writing keys to the tty/console, and the higher-level function emulates readline and its platform equivalents on top of nothing but that low-level stuff. I suspect you mean #2. In which case, again, look at how much code readline is. You're talking about duplicating all of that code, just to solve part of the problem (readline is still depending on your terminal to do a lot of the work, which you don't get in raw mode), on just one platform. >> A get_key_press function is the_easy_? part. On Unix, you use termios to >> put stdin into raw mode then select or blocking-read stdin; on Windows, >> you use the msvcrt module to call the wrappers around the console I/O >> functions. > >> But what do you think those functions return for, say, left arrow, or >> ctrl+end? (Even beyond non-character keys, even the characters don't >> necessarily show up as a single character, but as 1 or more bytes in the >> appropriate encoding (Unix) or 1 or more UTF-16 codes (Windows), so your >> state is less trivial than you're expecting?) > > Yes, there are a lot of small details to work out.? But I don't think any > of those are impossible to manage. Impossible, of course not. Just not remotely as easy as the part you called "the hard part". >> This is exactly why Python doesn't do any of that; it just uses >> readline >> (or libedit's readline-compat mode) on Unix, lets cmd.exe do its thing >> on Windows, etc. (This also makes it relatively easy for IDEs to hook >> the behavior and, e.g., make input give you a GUI window with a Tk text >> entry box.) > > Those will still work.? I don't think anyone is suggesting replacing all > of pythons console related code. You're missing the point. Those work because they use the services provided by the OS, the terminal, the readline library, etc. Trying to duplicate everything that all those layers do on every platform is a lot of work, and a bad idea. It's not about whether you replace anything with that new code or use it in parallel, it's that you're trying to build that new code in the first place. (But also, it's actually not true that they'll still work, at least not in the same problem. Try setting the tty to raw mode and then using readline and see what happens?) From tjreedy at udel.edu Fri Apr 4 10:54:07 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 04 Apr 2014 04:54:07 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: This proposal strikes me as conceptually disordered. A set is not ordered. An ordered set would be a sequence, and if you want a sequence, the best thing is to use a sequence object. A dict is a set of key,value pairs. If you want an ordered sequence of key, value pairs, use a sequence. OrderedDicts were added for situations where that is not possible because a pre-existing api requires a mapping object, even though you might prefer to use a sequence. Passing a sequence of key,value pairs is a case that requires a mapping. If a function attends to the order of key,value pairs it receives, it should receive a sequence of such. **kwds in a call is a substitute for writing out unordered key=value specifications. The function may or may not have **kargs in its signature and if it does, the kargs received may or may not be equal to the kwds passed. -- Terry Jan Reedy From p.f.moore at gmail.com Fri Apr 4 11:14:12 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 4 Apr 2014 10:14:12 +0100 Subject: [Python-ideas] Default value for input In-Reply-To: <1396591292.48940.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <20140401115842.GR16526@ando> <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396591292.48940.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: On 4 April 2014 07:01, Andrew Barnert wrote: > I suspect you mean #2. In which case, again, look at how much code readline is. You're talking about duplicating all of that code, just to solve part of the problem (readline is still depending on your terminal to do a lot of the work, which you don't get in raw mode), on just one platform. There's also pyreadline, which is a Pure-Python (plus ctypes) implementation of readline for Windows. AFAIK, it's pretty robust (IPython uses it), but the experience is so different from "normal" console behaviour that it's a but unnerving to a die-hard Windows console user like me. (The problem is that Python's core detects that readline is present and uses it, so if you install it, you get it in your normal interpreter whether you want it or not - I believe they were looking at fixing that in pyreadline, but it's been a while since I checked so I don't know if that's been done yet). Paul From random832 at fastmail.us Fri Apr 4 15:38:47 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 04 Apr 2014 09:38:47 -0400 Subject: [Python-ideas] Default value for input In-Reply-To: <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <20140401115842.GR16526@ando> <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <1396618727.3005.102752237.281AA1BF@webmail.messagingengine.com> On Thu, Apr 3, 2014, at 20:41, Andrew Barnert wrote: > A get_key_press function is the _easy_ part. On Unix, you use termios to > put stdin into raw mode then select or blocking-read stdin; on Windows, > you use the msvcrt module to call the wrappers around the console I/O > functions. A _good_ get_key_press function won't return bizarre character sequences for non-graphical keys in the first place. The msvcrt functions map a perfectly good high-level key event object into an information-losing abstraction designed to emulate a truly ancient MS-DOS API. "or 1 or more UTF-16 codes (Windows), so your state is less trivial than you're expecting?" Yeah, and just try distinguishing an arrow key from an _actual_ keypress of ? followed by a letter. From random832 at fastmail.us Fri Apr 4 15:47:13 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 04 Apr 2014 09:47:13 -0400 Subject: [Python-ideas] Default value for input In-Reply-To: <1396591292.48940.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <20140401115842.GR16526@ando> <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396591292.48940.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <1396619233.6062.102757189.1A26BE94@webmail.messagingengine.com> On Fri, Apr 4, 2014, at 2:01, Andrew Barnert wrote: > 1. The console class is basically a wrapper around readline or its > equivalent, in which case we have no need for get_key_press or any of the > other stuff you asked about. > > 2. The console class is just the low-level stuff that's implementable on > all platforms, like reading and writing keys to the tty/console, and the > higher-level function emulates readline and its platform equivalents on > top of nothing but that low-level stuff. > > I suspect you mean #2. In which case, again, look at how much code > readline is. You're talking about duplicating all of that code, just to > solve part of the problem (readline is still depending on your terminal > to do a lot of the work, which you don't get in raw mode), on just one > platform. A) You don't need any of it on windows, because the windows console's built-in editing functionality is good enough. Even this "default value" stuff wouldn't be hard - you just simulate a bunch of keypresses corresponding to the desired string. B) Any reasonable interface to the windows console won't use byte strings at all. From random832 at fastmail.us Fri Apr 4 15:50:21 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 04 Apr 2014 09:50:21 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> On Fri, Apr 4, 2014, at 4:54, Terry Reedy wrote: > **kwds in a call is a substitute for writing out unordered key=value > specifications. The function may or may not have **kargs in its > signature and if it does, the kargs received may or may not be equal to > the kwds passed. The problem is, there is no such thing as "writing out unordered key=value specifications" - anything that is "written out" has an order by its nature; the fact that this order is lost immediately is a poor abstraction. The desired use case is not to preserve the order of **kwds, it's to preserve the order of a sequence of key=value specifications. From ron3200 at gmail.com Fri Apr 4 16:20:29 2014 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 04 Apr 2014 10:20:29 -0400 Subject: [Python-ideas] Default value for input In-Reply-To: <1396591292.48940.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <20140401115842.GR16526@ando> <1396572081.39350.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396591292.48940.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: On 04/04/2014 02:01 AM, Andrew Barnert wrote: > From: Ron Adam > > Sent: Thursday, April 3, 2014 9:25 PM > > >> >On 04/03/2014 08:41 PM, Andrew Barnert wrote: >>> >> From: Ron Adam >>>> >>>If we could read the keyboard directly with a get_key_press >>>> >>> function, then >>>> >>>your request along with making curses cross platform, may be fairly >>>> >>>easy to do. >>>> >>>(The cross platform get_key_press function is the hard part.) >>> >> >>> >> Are you serious? >> > >> >Sure it's only a starting point, not a end solution by it self. I was >> >including the parts you you mention below, taking into account the platform >> >and console differences and returning a Unicode string. I wasn't thinking >> >of simply reading and returning the raw key codes. Just get the input >> >(including escape sequences) with out echoing it to the screen. > Again, that's easy to do on both platforms, but it doesn't even begin to > approach making the complete solution easy. I think you are thinking of a more complete solution than I am. >> >Probably the most straight forward way would be to import a console class, >> >and initiate an instance, then use methods on it to read the input and >> >write the output. That way all the platform stuff can be abstracted away. >> > >> >A higher level function that uses that class, would handle updating the >> >line buffer and write those updates out to the screen. (by calling methods >> >on the instance. > There are only two things you could mean here. > 1. The console class is basically a wrapper around readline or its > equivalent, in which case we have no need for get_key_press or any of > theother stuff you asked about. > 2. The console class is just the low-level stuff that's implementable > on all platforms, like reading and writing keys to the tty/console, and > the higher-level function emulates readline and its platform equivalents > on top of nothing but that low-level stuff. > I suspect you mean #2. In which case, again, look at how much code > readline is. You're talking about duplicating all of that code, just to > solve part of the problem (readline is still depending on your terminal to > do a lot of the work, which you don't get in raw mode), on just one platform. Which ever way would work best, but still result in a single API for all platforms. If number #1 can work, that would be fine, but I suspect there would be a lot of edge cases to resolve to get the wrappers to act alike, if it's even possible to do. I was thinking of just having the basics of #2 in place. Then wait a bit to see how it's used and not try to duplicate readline. Not everything needs the full features of readline. And it may be possible to switch to simple (option #2) input mode temporarily and then back to readline or it equivalent mode. (?) In any case. If you still think it's a bad idea, I'm fine with that. >>> >> This is exactly why Python doesn't do any of that; it just uses >>> >>readline >>> >> (or libedit's readline-compat mode) on Unix, lets cmd.exe do its thing >>> >> on Windows, etc. (This also makes it relatively easy for IDEs to hook >>> >> the behavior and, e.g., make input give you a GUI window with a Tk text >>> >> entry box.) >> > >> >Those will still work. I don't think anyone is suggesting replacing all >> >of pythons console related code. > You're missing the point. Those work because they use the services > provided by the OS, the terminal, the readline library, etc. Trying to > duplicate everything that all those layers do on every platform is a lot of > work, and a bad idea. It's not about whether you replace anything with that > new code or use it in parallel, it's that you're trying to build that new > code in the first place. > (But also, it's actually not true that they'll still work, at least not > in the same problem. Try setting the tty to raw mode and then using > readline and see what happens?) Well.. don't do that. ;-) Cheers, Ron From mertz at gnosis.cx Fri Apr 4 18:04:07 2014 From: mertz at gnosis.cx (David Mertz) Date: Fri, 4 Apr 2014 12:04:07 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: I confess that after reading this thread, and a number of related past ones, I'm still not certain exactly what problem all of this is needed to solve. ISTM that if one has a special function calling requirement to pass in an ordered collection of key/value pairs, one can already just use a special and available call signature for your function: def myfunc(a, b, *keyvals): od = OrderedDict(keyvals) # ... do other stuff Call this like: value = myfunc(foo, bar, ('a',1), ('z',2), ('b',3)) Yes, it's a slightly special form of the calling convention, but it does something slightly special with its key/val-like arguments, so that seems like a reasonable tradeoff. The solution is purely local to the writer of the function who needs this. Even if you have an existing OrderedDict that you want to pass in, you can use that like: value = myfunc(foo, bar, *myOD.items()) Of course, if you want to be picky, you could stick in a check at the top of your function definition: assert all(isinstance(x, tuple) and len(x)==2 for x in keyvals) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Fri Apr 4 19:15:14 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 04 Apr 2014 13:15:14 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: <1396631714.4383.102836165.76A6595B@webmail.messagingengine.com> On Fri, Apr 4, 2014, at 12:04, David Mertz wrote: > I confess that after reading this thread, and a number of related past > ones, I'm still not certain exactly what problem all of this is needed to > solve. ISTM that if one has a special function calling requirement to > pass > in an ordered collection of key/value pairs, one can already just use a > special and available call signature for your function: People want to be able to pass them in order __with the a=1, b=2 syntax__. Because the syntax you're proposing looks like grit on my monitor. The most commonly cited desired use case is for the OrderedDict constructor itself. From abarnert at yahoo.com Fri Apr 4 19:46:37 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 4 Apr 2014 10:46:37 -0700 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396631714.4383.102836165.76A6595B@webmail.messagingengine.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> <1396631714.4383.102836165.76A6595B@webmail.messagingengine.com> Message-ID: <8654974B-938A-4DE2-BE5A-88837971627D@yahoo.com> On Apr 4, 2014, at 10:15, random832 at fastmail.us wrote: > On Fri, Apr 4, 2014, at 12:04, David Mertz wrote: >> I confess that after reading this thread, and a number of related past >> ones, I'm still not certain exactly what problem all of this is needed to >> solve. ISTM that if one has a special function calling requirement to >> pass >> in an ordered collection of key/value pairs, one can already just use a >> special and available call signature for your function: > > People want to be able to pass them in order __with the a=1, b=2 > syntax__. Because the syntax you're proposing looks like grit on my > monitor. > > The most commonly cited desired use case is for the OrderedDict Every time this comes up, that's the only good example anyone comes up with. It's obviously a special case. Besides being self-referential, a solution to that one case would automatically drastically improve the workarounds for any other case that anyone else later finds. If we need a general-purpose solution, then it has to be general-purpose. It has to work with dynamically-identified callables, forwarding wrappers, etc. And that's hard. If we only need OrderedDict to work, there are a lot fewer requirements. From abarnert at yahoo.com Fri Apr 4 19:51:26 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 4 Apr 2014 10:51:26 -0700 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: <458C83D9-C21E-4378-BBD0-8B7C64F14C74@yahoo.com> On Apr 4, 2014, at 9:04, David Mertz wrote: > def myfunc(a, b, *keyvals): > od = OrderedDict(keyvals) > # ... do other stuff > > Call this like: > > value = myfunc(foo, bar, ('a',1), ('z',2), ('b',3)) Or, possibly more readably: def myfunc(a, b, *keydicts): od = OrderedDict((k, v) for keydicts in keydicts for k, v in keydict.items()) value = myfunc(foo, bar, {'a': 1}, {'z': 2}) Not as clean as passing a single dict literal, or a sequence of keywords, but still clearly a sequence of key-value pairs rather than just a sequence of sequences... From tjreedy at udel.edu Sat Apr 5 05:16:55 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 04 Apr 2014 23:16:55 -0400 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: On 4/4/2014 9:50 AM, random832 at fastmail.us wrote: > On Fri, Apr 4, 2014, at 4:54, Terry Reedy wrote: >> **kwds in a call is a substitute for writing out unordered key=value >> specifications. The function may or may not have **kargs in its >> signature and if it does, the kargs received may or may not be equal to >> the kwds passed. > > The problem is, there is no such thing as "writing out unordered > key=value specifications" - anything that is "written out" has an order > by its nature; Before being written, the key=value specification is unordered. The order introduced by writing it out in linear text is spurious. > the fact that this order is lost immediately is a poor > abstraction. The desired use case is not to preserve the order of > **kwds, it's to preserve the order of a sequence of key=value > specifications. Wanting to preserve spurious order is wrong. What I said before is that if one has meaningful order, one should better use an ordered sequence, both in the definition and call of the function, rather than an unordered (or spuriously ordered) mapping. -- Terry Jan Reedy From rosuav at gmail.com Sat Apr 5 05:50:37 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 Apr 2014 14:50:37 +1100 Subject: [Python-ideas] Retain string form of AST Numbers Message-ID: Currently, parsing a Python script into an AST discards a huge amount of information. Some of that (line and column information) is retained as metadata, but a lot isn't. One of the potentially-useful things that's currently dropped is the string form of numbers: >>> print(ast.dump(ast.parse("x = 1 + 2.000 + 3.0j"))) Module(body=[Assign(targets=[Name(id='x', ctx=Store())], value=BinOp(left=BinOp(left=Num(n=1), op=Add(), right=Num(n=2.0)), op=Add(), right=Num(n=3j)))]) The three Num nodes have an n with the already-constructed int/float/complex. At this point, it's impossible to return to the original string that was entered; any rounding will have already taken place. If the Num nodes also had a .str member with the unparsed string ("1", "2.000", and "3.0j"), it would be possible to influence their interpretation with an AST transform - for instance, turning them into decimal.Decimal construction calls, or wrapping them in something that automatically keeps track of significant figures (which would mean that "2.000" is considered more precise than a mere "2.0" or "2."). It shouldn't be too difficult to do, and it could simply be ignored in most cases; any sort of change to the actual number could leave the original string out of sync with it, which isn't a problem. Thoughts? ChrisA From dickinsm at gmail.com Sat Apr 5 10:09:55 2014 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 5 Apr 2014 09:09:55 +0100 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: On Sat, Apr 5, 2014 at 4:50 AM, Chris Angelico wrote: > If the Num nodes also had a .str member with the unparsed string ("1", > "2.000", and "3.0j"), it would be possible to influence their > interpretation with an AST transform - for instance, turning them into > decimal.Decimal construction calls, or wrapping them in something that > automatically keeps track of significant figures (which would mean > that "2.000" is considered more precise than a mere "2.0" or "2."). > > It shouldn't be too difficult to do, and it could simply be ignored in > most cases; any sort of change to the actual number could leave the > original string out of sync with it, which isn't a problem. > > Thoughts? > I like it. Apart from the AST transformation possibilities opened up, I have some AST-based style checkers that would have benefited from access to the raw numeric strings. So as long as there's no significant adverse affect on total compilation time or space requirements, +1 from me. Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From dickinsm at gmail.com Sat Apr 5 10:10:51 2014 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 5 Apr 2014 09:10:51 +0100 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: On Sat, Apr 5, 2014 at 9:09 AM, Mark Dickinson wrote: > So as long as there's no significant adverse affect on total compilation > time or space requirements, +1 from me. > I can't quite believe I wrote that. adverse *effect*! -- Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Apr 5 11:25:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 5 Apr 2014 19:25:38 +1000 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: On 5 April 2014 18:09, Mark Dickinson wrote: > On Sat, Apr 5, 2014 at 4:50 AM, Chris Angelico wrote: >> >> If the Num nodes also had a .str member with the unparsed string ("1", >> "2.000", and "3.0j"), it would be possible to influence their >> interpretation with an AST transform - for instance, turning them into >> decimal.Decimal construction calls, or wrapping them in something that >> automatically keeps track of significant figures (which would mean >> that "2.000" is considered more precise than a mere "2.0" or "2."). >> >> It shouldn't be too difficult to do, and it could simply be ignored in >> most cases; any sort of change to the actual number could leave the >> original string out of sync with it, which isn't a problem. >> >> Thoughts? > > > I like it. Apart from the AST transformation possibilities opened up, I have > some AST-based style checkers that would have benefited from access to the > raw numeric strings. > > So as long as there's no significant adverse affect on total compilation > time or space requirements, +1 from me. Also somewhat tangentially related to this issue about being able to render particular args differently in a signature: http://bugs.python.org/issue16801 I think the discussion on http://bugs.python.org/issue10769 suggests a broader philosophical discussion may be appropriate, since it makes a big difference to what is in scope for the ast module if its purpose is expanded from "easier to manipulate intermediate representation between source and bytecode" to "supports lossless source to source transformations". Articles like http://time-loop.tumblr.com/post/47664644/python-ast-preserving-whitespace-and-comments suggest to me that we may need to start pushing people *away* from the ast module and add a lib2to3 inspired API specifically for lossless transformations based on the *actual* grammar of the running Python version (the grammar 2to3 itself uses is a bit mixed up, because it needs to tolerate Python 2 code as input, and it currently doesn't handle "yield from" expressions - the latter isn't really an issue in practice for 2/3 transition related use cases, since hybrid 2/3 code can't use "yield from" anyway). (A Google search for "Python AST preserve comments" provides other interesting links on the topic of source->source transformations that aren't lossy the way those based on the ast module inevitably are) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Sat Apr 5 13:53:34 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 5 Apr 2014 22:53:34 +1100 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: On Sat, Apr 5, 2014 at 8:25 PM, Nick Coghlan wrote: > Articles like http://time-loop.tumblr.com/post/47664644/python-ast-preserving-whitespace-and-comments > suggest to me that we may need to start pushing people *away* from the > ast module and add a lib2to3 inspired API specifically for lossless > transformations based on the *actual* grammar of the running Python > version Ooh that looks promising. I might try to knock up a recipe for some sort of "interactive Python in decimal mode", which would cover one of the important cases for which people ask for decimal literals. It'd not be perfect without some monkey-patch for int/int -> float, unless simply every non-complex number becomes a Decimal. Or if I don't get around to it, maybe someone else can? ChrisA From rymg19 at gmail.com Sat Apr 5 15:32:45 2014 From: rymg19 at gmail.com (Ryan) Date: Sat, 05 Apr 2014 08:32:45 -0500 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: <6980302d-8c0f-4be2-802d-1a710610f008@email.android.com> I could probably try that if you give a slightly clearer description. Chris Angelico wrote: >On Sat, Apr 5, 2014 at 8:25 PM, Nick Coghlan >wrote: >> Articles like >http://time-loop.tumblr.com/post/47664644/python-ast-preserving-whitespace-and-comments >> suggest to me that we may need to start pushing people *away* from >the >> ast module and add a lib2to3 inspired API specifically for lossless >> transformations based on the *actual* grammar of the running Python >> version > >Ooh that looks promising. I might try to knock up a recipe for some >sort of "interactive Python in decimal mode", which would cover one of >the important cases for which people ask for decimal literals. It'd >not be perfect without some monkey-patch for int/int -> float, unless >simply every non-complex number becomes a Decimal. > >Or if I don't get around to it, maybe someone else can? > >ChrisA >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 5 16:03:22 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 6 Apr 2014 01:03:22 +1100 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: <6980302d-8c0f-4be2-802d-1a710610f008@email.android.com> References: <6980302d-8c0f-4be2-802d-1a710610f008@email.android.com> Message-ID: On Sun, Apr 6, 2014 at 12:32 AM, Ryan wrote: > I could probably try that if you give a slightly clearer description. Currently, you can fire up IDLE or interactive Python and use it as a super calculator: >>> 1 + 2 3 And when you use it with non-integers, you get floats: >>> 0.1 + 0.2 0.30000000000000004 We could turn this into a Decimal calculator with two important changes: 1) Every instance of a number in the code must become a Decimal (which really means wrapping every number with Decimal("...") and importing Decimal from decimal) 2) Monkey-patch Decimal.__repr__ = Decimal.__str__ to make the display tidy The latter might be better done by subclassing Decimal and putting something into the builtins, but that's implementation detail (now that Decimal is implemented in C, its attributes can't be set). As long as you don't need multiple contexts or anything complicated like that, this should in theory work. (Changing the default context shouldn't break anything AFAICT.) It'd then be usable in the same way that REXXTry is: a convenient calculator that's backed by a full programming language. It'd be better than REXXTry, in fact, as limitations on REXX's 'INTERPRET' command mean you can't define functions that way (though you can call functions defined in REXXTry itself - I had my own Enhanced REXXTry with gobs of mathematical utility functions). If that can be done without any core language changes, that would be awesome. ChrisA From aj at erisian.com.au Sat Apr 5 18:07:00 2014 From: aj at erisian.com.au (Anthony Towns) Date: Sun, 6 Apr 2014 02:07:00 +1000 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: On 2 April 2014 16:54, Philipp A. wrote: > i have this one floating around: > https://gist.github.com/flying-sheep/9929117 I've rejigged this a bit more, and put it up on pypi fwiw: https://pypi.python.org/pypi/odictliteral Cheers, aj -- Anthony Towns From storchaka at gmail.com Sat Apr 5 19:39:06 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 05 Apr 2014 20:39:06 +0300 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: 05.04.14 06:50, Chris Angelico ???????(??): > Currently, parsing a Python script into an AST discards a huge amount > of information. Some of that (line and column information) is retained > as metadata, but a lot isn't. One of the potentially-useful things > that's currently dropped is the string form of numbers: > >>>> print(ast.dump(ast.parse("x = 1 + 2.000 + 3.0j"))) > Module(body=[Assign(targets=[Name(id='x', ctx=Store())], > value=BinOp(left=BinOp(left=Num(n=1), op=Add(), right=Num(n=2.0)), > op=Add(), right=Num(n=3j)))]) > > The three Num nodes have an n with the already-constructed > int/float/complex. At this point, it's impossible to return to the > original string that was entered; any rounding will have already taken > place. > > If the Num nodes also had a .str member with the unparsed string ("1", > "2.000", and "3.0j"), it would be possible to influence their > interpretation with an AST transform - for instance, turning them into > decimal.Decimal construction calls, or wrapping them in something that > automatically keeps track of significant figures (which would mean > that "2.000" is considered more precise than a mere "2.0" or "2."). > > It shouldn't be too difficult to do, and it could simply be ignored in > most cases; any sort of change to the actual number could leave the > original string out of sync with it, which isn't a problem. > > Thoughts? What about strings? >>> print(ast.dump(ast.parse(r"x = '''\x0a\u000d''' r'x'"))) Module(body=[Assign(targets=[Name(id='x', ctx=Store())], value=Str(s='\n\rx'))]) Tuples? >>> print(ast.dump(ast.parse("x = (1, 2,)"))) Module(body=[Assign(targets=[Name(id='x', ctx=Store())], value=Tuple(elts=[Num(n=1), Num(n=2)], ctx=Load()))]) From ericsnowcurrently at gmail.com Sat Apr 5 19:51:16 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 5 Apr 2014 11:51:16 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: <1396590161.64397.YahooMailNeo@web181003.mail.ne1.yahoo.com> References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396590161.64397.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: On Apr 3, 2014 11:42 PM, "Andrew Barnert" wrote: > > From: Eric Snow > > Sent: Thursday, April 3, 2014 8:03 PM > > > > On Thu, Apr 3, 2014 at 7:50 PM, Andrew Barnert wrote: > > > [snip] > > > So you are saying that it is not uncommon to use "perfect passthrough" > > decorators/wrappers you do not control. > > Yes. Just my first example, partial, is ridiculously common, and it's a wrapper I don't control. Half the servers, GUIs, and other callback-based programs I've worked on need partial functions, and get them either by using partial, or by using lambda functions instead--which has an equivalent and parallel problem that, come to think of it, still requires a solution too... > > However, what we _don't_ know is how common it would be to use such wrappers with functions that need ordered kwargs. The prototypical case of using partial on GUI or server callbacks obviously doesn't apply there--in a Tkinter GUI, your callback functions almost never take kwargs, and, if they did, it would probably be just to pass the dict as a widget configure dict or something like that, which couldn't possibly care about the order. But really, _no_ example that works today could apply; it's not possible to write functions that care about their keyword order, so we can only try to guess whether the potential use cases for them could involve wrapping them or not. > > > Regardless, the workarounds are the same. Likewise they add the same > > complication. Perhaps it does favor the decorator syntax over the > > __kworder__ local variable. > > I'm not sure it favors either. Either way, every function that returns a forwarding wrapper will need to be changed to apply the workaround. The workaround itself may be easier, or at least prettier, with the decorator, but the hard part isn't the workaround itself (as I showed, it could just be a simple on-line change), but figuring out which wrappers need it, and possibly copying and forking them into your own code, so you can apply the workaround to each one. Right. That's why a ordered-by-default approach is favorable. Providing an opt-out mechanism should satisfy what I still think are uncommon cases. > > [snip] > > > Looking into the implications on inspect.signature, et al. is on my > > to-do list. :) > > After a little more thought, what's really needed is a consistent way to represent ordered-kwargs parameters in *documentation*. > > Once you have that, whatever it is, inspect.signature should be simple. For example, if you use kind=VAR_ORDERED_KEYWORD, then it's just a matter of making Parameter.__str__ return whatever-it-is that the docs would have instead of returning the normal **{name}. > > This does give a little more weight to proposals that do something visible in the signature itself--the ***okwargs, or an annotation, or whatever. Makes sense. > But I think all of them have enough down-sides compared to your more favored versions that it wouldn't be nearly enough weight to shift things. Perhaps. At this point nothing much has been settled. My gut says you're right, but we'll see. > > Guido's concern was that in some uncommon cases the difference in > > performance (both memory and speed) would render OrderedDict > > undesirable for **kwargs. By storing the initial order on a dict > > attribute the only overhead would be 1 pointer per dict plus the size > > of the container. > > First, after reading the messages you linked, I don't think your characterizing Guido's concerns fairly. > > His first message is not about uncommon cases at all, it's about the fact that every function call (at least every one that uses kwargs) will slow down noticeably: "My main concern is speed -- since most code doesn't need it and function calls are already slow (and obviously very common :-) it would be a shame if this slowed down function calls that don't need it noticeably." In the best case the slowdown on each call would be the cost of checking a flag and not even for most calls. In the worst case I image it will be the cost of allocating a tuple/list (a free list would help) and populating it, while again this would only impact functions that define **kwargs). > It's hard to predict how much of a slowdown a nice C OrderedDict would cause here, and how much less of a slowdown just creating and using __order__ would cause. But from a quick simulation in pure Python, it looks like you're just turning a 2.7x slowdown into a 2.2x slowdown--better, but still clearly noticeable, and therefore not acceptable. At one point I benchmarked my C OrderedDict (issue #16991) against dict. For most operations it was the same, including all non-mutating operations. If I recall correctly, iteration was even faster (due to traversing a linked list rather than a sparse hash table). For mutating operations the overhead resulted in up to a 3x slowdown. > > This means you need some way to make sure functions that don't ask for ordered kwargs don't get it (unless you can actually make it not noticeably slower). The fact that this only affects functions that define **kwargs minimizes the scope of the impact. > I think all of your solutions (including storing __kworder__) will be noticeably slower; there's just no way around the fact that maintaining a dict and its order and iterating it in custom order takes longer than maintaining a dict and iterating it in natural order. I'm pretty sure this only matters in the **kwargs unpacking case. Even then, if we go with OrderedDict unpacking into the new kwargs is only marginally slower than dict. I do not know the impact of unpacking into a secondary order-preserving container in addition to into dict, but I expect it will not be much worse. Again this should only impact cases where the function defines **kwargs. > Trying to disguise the ordering info, or make it not quite as slow, or make it opt-out instead of opt-in, etc., doesn't solve this problem. Only finding a way to do nothing at all when nobody wants it solves this problem. At which point you might as well do the simplest thing--use OrderedDict--for cases where people _do_ want it. > > And that comes back to my original point: Where does someone want order? When they explicitly ask for it, but also when wrapping a function that asks for it in a generic wrapper. And that's the hard part; inside the wrapper (in some library code), you don't know to ask for it; outside the wrapper (in your app), there's no way to do it. I agree that this use case ("perfect kwargs preserving" wrappers) is more complicated under the circumstances you described. However, don't discount the options on the table that do not present those circumstances. > > Meanwhile, the "uncommon" cases (which you elsewhere call "rare", "extreme", and "edge cases") are a different story. I don't think Guido thinks they're very uncommon. He says, "I write code regularly that..." saves kwargs for later, and possibly even piles other stuff into it. And I will argue that while perhaps not uncommon for Guido, they are uncommon for most people. > His primary issue here is that having anything other than a plain dict is wrong and confusing. Hmm. I did not get that, much less that it is his primary concern. I took away that he was concerned about the performance impact in the case where kwargs outlives the function. Plus, this is Python. As long as it quacks like a dict we're fine. With OrderedDict it's even a subclass of dict. > Secondarily, he also mentions "possible" performance problems. Here is where the memory comes in. Increasing the memory of every kwargs dict by 25% instead of 75% is nice, but I don't think it solves the memory problem, and I don't think the memory problem is the main issue anyway. I agree it's not the main issue. > > This seems to rule out any solution that gives you a modified kwargs dict when you don't ask for it, whether it's an OrderedDict or a dict with an __order__ attribute or anything else. Due the memory issues? Memory is only an issue in cases where the resultant kwargs is going to be extremely large, which I still say is uncommon. Furthermore, using dict.__order__ means that even then it's an issue only with an extremely large number of kwargs passed in since __order__ wouldn't grow with the dict. And opt-out would allow people to avoid the issue in those cases. > I now understand why you proposed the separate __kworder__ magic parameter solution. But really, given the first problem, any solution that adds _any_ noticeable overhead when you don't ask for it is already ruled out, so trying to put the extra overhead somewhere other than the dict itself doesn't help. You need to actually not create it. There shouldn't be any noticeable overhead so this point is somewhat moot. If there is then this proposal is on pretty shaky ground. > > Anyway, after a little more back-burner thought, I actually did think of some magic that might work. But I don't think you, or anyone else, will like it. I certainly don't. It's good to look at all the options. And this is python-ideas after all. :) > At compile time, any code object that has a call to something accessed via a closure cell gets a calls-local flag. (This should catch all decorators and other wrapper-generators, but many false positives as well.) At runtime, when building a function from a code object with the calls-local flag, if any of the __closure__ members are functions with the order-kwargs flag, the new function also gets the flag as well. (This eliminates most of the false positives, without eliminating any actual wrappers unless they're designed pathologically. And if they are pathological, well, then they lose kwargs ordering.) That's all only a couple lines of code in the compiler and interpreter. Tada? Not sure; the idea is too horrible and non-Pythonic to think through all the way. Ultimately, this > design would mean that all local functions get ordered kwargs, but there's an optimization that usually eliminates the order when you don't need it... which is just wrong. That's not so bad. I don't think something like that would be needed though. This is definitely an implementation detail. It would be important to minimize any further complication to function-call code. Otherwise the implementation will be a hard sell. Sort of relatedly, one thing to consider when adding language features is the impact they will have on other implementations. That relates almost entirely on the explicit additions to the language spec so we need to be careful about favoring solutions strictly for implementation concerns. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sat Apr 5 20:13:00 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 5 Apr 2014 12:13:00 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> Message-ID: On Apr 4, 2014 2:54 AM, "Terry Reedy" wrote: > > This proposal strikes me as conceptually disordered. A set is not ordered. An ordered set would be a sequence, and if you want a sequence, the best thing is to use a sequence object. A dict is a set of key,value pairs. If you want an ordered sequence of key, value pairs, use a sequence. OrderedDicts were added for situations where that is not possible because a pre-existing api requires a mapping object, even though you might prefer to use a sequence. With OrderedDict it's about having a mapping and the order of the keys in one data structure. A sequence of pairs does not an ordered mapping make, though they may contain the same data. > > Passing a sequence of key,value pairs is a case that requires a mapping. If a function attends to the order of key,value pairs it receives, it should receive a sequence of such. > > **kwds in a call is a substitute for writing out unordered key=value specifications. The function may or may not have **kargs in its signature and if it does, the kargs received may or may not be equal to the kwds passed. As someone else noted, you write out an *ordered* key=value specification and that information is lost. In the ** unpacking case, any order information in the unpacked object is likewise lost. This proposal is all about preserving that order information in some way without requiring the caller to extract the order information manually and passing it in separately. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sat Apr 5 20:21:58 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 5 Apr 2014 12:21:58 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: On Apr 4, 2014 7:50 AM, wrote: > The problem is, there is no such thing as "writing out unordered > key=value specifications" - anything that is "written out" has an order > by its nature; the fact that this order is lost immediately is a poor > abstraction. The desired use case is not to preserve the order of > **kwds, it's to preserve the order of a sequence of key=value > specifications. Right. However, it is also important to preserve the order, if any, found in containers unpacked (via **) into a call. The possibility of "perfect kwargs preserving" wrappers (thanks Andrew) blurs the distinction. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sat Apr 5 20:49:24 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 5 Apr 2014 12:49:24 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: On Apr 4, 2014 10:04 AM, "David Mertz" wrote: > I confess that after reading this thread, and a number of related past ones, I'm still not certain exactly what problem all of this is needed to solve. Yeah, it won't be worth it without justification. :) I'll spell it out in the PEP, but basically it boils down to these reasons: * Helpful in debugging (from Raymond and others). * Controlling object presentation. * Anywhere you want to set iteration order and name/value in a single call (from Nick). + Factories for ordered types. + Serializable objects where order matters. * Keyword argument priority. I have a few more less concrete use cases as well. OrderedDict is the first example everyone cites, but it's simply the most obvious. Furthermore, it's a rather self-referential example. :) As I said, I'll make the full case in the PEP, which I hope to get out today. > ISTM that if one has a special function calling requirement to pass in an ordered collection of key/value pairs, one can already just use a special and available call signature for your function: > > def myfunc(a, b, *keyvals): > od = OrderedDict(keyvals) > # ... do other stuff > > Call this like: > > value = myfunc(foo, bar, ('a',1), ('z',2), ('b',3)) > > Yes, it's a slightly special form of the calling convention, but it does something slightly special with its key/val-like arguments, so that seems like a reasonable tradeoff. The solution is purely local to the writer of the function who needs this. That's clever. However, it is basically the same as the status quo: if you want to preserve order you cannot use keyword arguments, including ** unpacking, even though they are the natural way to do it. For existing functions, preserving order means changing code even though the order is already available. The point of my proposal is to teach the interpreter to preserve the order. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sat Apr 5 21:46:48 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 5 Apr 2014 13:46:48 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: On Apr 4, 2014 9:17 PM, "Terry Reedy" wrote: > Before being written, the key=value specification is unordered. The order introduced by writing it out in linear text is spurious. [snip] > Wanting to preserve spurious order is wrong. What I said before is that if one has meaningful order, one should better use an ordered sequence, both in the definition and call of the function, rather than an unordered (or spuriously ordered) mapping. Sometimes the order is not spurious. That's the point of making sure we enumerate valid use cases for this proposal. Otherwise the cost to the language isn't worth it. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Apr 5 22:31:44 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 05 Apr 2014 16:31:44 -0400 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: <6980302d-8c0f-4be2-802d-1a710610f008@email.android.com> Message-ID: On 4/5/2014 10:03 AM, Chris Angelico wrote: > On Sun, Apr 6, 2014 at 12:32 AM, Ryan wrote: >> I could probably try that if you give a slightly clearer description. > > Currently, you can fire up IDLE or interactive Python and use it as a > super calculator: > >>>> 1 + 2 > 3 > > And when you use it with non-integers, you get floats: > >>>> 0.1 + 0.2 > 0.30000000000000004 > > We could turn this into a Decimal calculator with two important changes: > 1) Every instance of a number in the code must become a Decimal (which > really means wrapping every number with Decimal("...") and importing > Decimal from decimal) > 2) Monkey-patch Decimal.__repr__ = Decimal.__str__ to make the display tidy > > The latter might be better done by subclassing Decimal and putting > something into the builtins, but that's implementation detail (now > that Decimal is implemented in C, its attributes can't be set). > > If that can be done without any core language changes, that would be awesome. One could certainly do something with Idle without changing Python. Idle has an extension mechanism that operates by modifying runtime structures. I believe it is mainly used to add menu entries for functions that operate on the edit buffer wrapped by an editor window. Thus it should be possible to 'decimalize' a file by adding an import (or subclass definition) at the top of the buffer and wrap literals with D("..."). I don't think that the existing hooks are enough to modify the behavior of the Shell. For that, one could write python code to permanently patch the Shell to add a decimal mode. It would need to send initialization code to each user subprocess and wrap literals as above, either as typed or before sending. When I have more time than I do now, I would be willing to share what I know about how either could be done. -- Terry Jan Reedy From rymg19 at gmail.com Sat Apr 5 22:42:58 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sat, 5 Apr 2014 15:42:58 -0500 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: <6980302d-8c0f-4be2-802d-1a710610f008@email.android.com> Message-ID: On Sat, Apr 5, 2014 at 9:03 AM, Chris Angelico wrote: > On Sun, Apr 6, 2014 at 12:32 AM, Ryan wrote: > > I could probably try that if you give a slightly clearer description. > > Currently, you can fire up IDLE or interactive Python and use it as a > super calculator: > > >>> 1 + 2 > 3 > > And when you use it with non-integers, you get floats: > > >>> 0.1 + 0.2 > 0.30000000000000004 > > We could turn this into a Decimal calculator with two important changes: > 1) Every instance of a number in the code must become a Decimal (which > really means wrapping every number with Decimal("...") and importing > Decimal from decimal) > Simple enough. > 2) Monkey-patch Decimal.__repr__ = Decimal.__str__ to make the display tidy > I actually found a module to do that: https://pypi.python.org/pypi/forbiddenfruit/0.1.1. Dangerous but awesome. > The latter might be better done by subclassing Decimal and putting > something into the builtins, but that's implementation detail (now > that Decimal is implemented in C, its attributes can't be set). > > As long as you don't need multiple contexts or anything complicated > like that, this should in theory work. (Changing the default context > shouldn't break anything AFAICT.) It'd then be usable in the same way > that REXXTry is: a convenient calculator that's backed by a full > programming language. It'd be better than REXXTry, in fact, as > limitations on REXX's 'INTERPRET' command mean you can't define > functions that way (though you can call functions defined in REXXTry > itself - I had my own Enhanced REXXTry with gobs of mathematical > utility functions). > > If that can be done without any core language changes, that would be > awesome. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Apr 5 23:38:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Apr 2014 07:38:14 +1000 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: On 6 Apr 2014 05:12, "Eric Snow" wrote: > > On Apr 4, 2014 10:04 AM, "David Mertz" wrote: > > I confess that after reading this thread, and a number of related past ones, I'm still not certain exactly what problem all of this is needed to solve. > > Yeah, it won't be worth it without justification. :) I'll spell it out in the PEP, but basically it boils down to these reasons: > > * Helpful in debugging (from Raymond and others). > * Controlling object presentation. > * Anywhere you want to set iteration order and name/value in a single call (from Nick). > + Factories for ordered types. > + Serializable objects where order matters. In the context of serialisation, one key lesson we have learned is that arbitrary ordering is a problem when you want to minimise spurious diffs, and sorting isn't a simple solution. Tools like doctest don't tolerate spurious diffs at all, but are often amenable to a sorting based answer. The cases where it would be highly desirable to be able use keyword arguments to control the order of display of a collection of key value pairs are ones like: * printing out key:value pairs in CLI output * mapping semantic names to column order in a CSV * serialising attributes and elements in particular orders in XML * serialising map keys in particular orders in human readable formats like JSON and YAML (particularly when they're going to be placed under source control) These *can* all be done today, but *not* by using keyword arguments. In my view, the problem to be addressed is that keyword arguments *look* like they should work for these cases, because they have a definite order in the source code. The only reason they don't work is because the interpreter throws that ordering information away. It's a textbook case of a language feature becoming an attractive nuisance in some circumstances: the simple and obvious solution for the above use cases *doesn't actually work* for reasons that aren't obviously clear if you don't have a firm grasp of Python's admittedly complicated argument handling. The simplest way out that I can see? Just make it work, even if that means also preserving the order of arbitrary keyword arguments in cases that *dont* need it. Deciding whether or not to provide a way to opt in to dropping the order info for speed and memory reasons then becomes a separate optimisation discussion *after* the current usability trap has been addressed. Cheers, Nick. > * Keyword argument priority. > > I have a few more less concrete use cases as well. OrderedDict is the first example everyone cites, but it's simply the most obvious. Furthermore, it's a rather self-referential example. :) As I said, I'll make the full case in the PEP, which I hope to get out today. > > > ISTM that if one has a special function calling requirement to pass in an ordered collection of key/value pairs, one can already just use a special and available call signature for your function: > > > > def myfunc(a, b, *keyvals): > > od = OrderedDict(keyvals) > > # ... do other stuff > > > > Call this like: > > > > value = myfunc(foo, bar, ('a',1), ('z',2), ('b',3)) > > > > Yes, it's a slightly special form of the calling convention, but it does something slightly special with its key/val-like arguments, so that seems like a reasonable tradeoff. The solution is purely local to the writer of the function who needs this. > > That's clever. However, it is basically the same as the status quo: if you want to preserve order you cannot use keyword arguments, including ** unpacking, even though they are the natural way to do it. For existing functions, preserving order means changing code even though the order is already available. The point of my proposal is to teach the interpreter to preserve the order. > > -eric > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 5 23:55:03 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 6 Apr 2014 07:55:03 +1000 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: On Sun, Apr 6, 2014 at 3:39 AM, Serhiy Storchaka wrote: > What about strings? > >>>> print(ast.dump(ast.parse(r"x = '''\x0a\u000d''' r'x'"))) > Module(body=[Assign(targets=[Name(id='x', ctx=Store())], > value=Str(s='\n\rx'))]) > > Tuples? > >>>> print(ast.dump(ast.parse("x = (1, 2,)"))) > Module(body=[Assign(targets=[Name(id='x', ctx=Store())], > value=Tuple(elts=[Num(n=1), Num(n=2)], ctx=Load()))]) I don't have any concrete use cases for those, but the same could be done for every type of node as a general mechanism for recreating a file more exactly. Nick linked to a tracker issue with some discussion on docstrings, which could benefit from that. Or maybe it would be better to avoid the AST altogether and do a source-level translation, as mentioned in the blog post he also linked to. ChrisA From guido at python.org Sun Apr 6 00:09:10 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 5 Apr 2014 15:09:10 -0700 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: Whoops, caps lock error. Didn't mean to shout. :-) On Saturday, April 5, 2014, Guido van Rossum wrote: > Lib2to3 has its own parser and tree so it can reconstruct the file, with > comments. AST JUST THROWS AWAY TOO MUCH. > > On Saturday, April 5, 2014, Chris Angelico > > wrote: > >> On Sun, Apr 6, 2014 at 3:39 AM, Serhiy Storchaka >> wrote: >> > What about strings? >> > >> >>>> print(ast.dump(ast.parse(r"x = '''\x0a\u000d''' r'x'"))) >> > Module(body=[Assign(targets=[Name(id='x', ctx=Store())], >> > value=Str(s='\n\rx'))]) >> > >> > Tuples? >> > >> >>>> print(ast.dump(ast.parse("x = (1, 2,)"))) >> > Module(body=[Assign(targets=[Name(id='x', ctx=Store())], >> > value=Tuple(elts=[Num(n=1), Num(n=2)], ctx=Load()))]) >> >> I don't have any concrete use cases for those, but the same could be >> done for every type of node as a general mechanism for recreating a >> file more exactly. Nick linked to a tracker issue with some discussion >> on docstrings, which could benefit from that. Or maybe it would be >> better to avoid the AST altogether and do a source-level translation, >> as mentioned in the blog post he also linked to. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > -- > --Guido van Rossum (on iPad) > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Apr 6 00:11:20 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 6 Apr 2014 08:11:20 +1000 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: On Sun, Apr 6, 2014 at 8:09 AM, Guido van Rossum wrote: > Whoops, caps lock error. Didn't mean to shout. :-) Heh, glad I hadn't offended you there :) I've put "Learn the 2to3 parser and use it for source code transformation" (with Nick's link) in my TODO. No telling when it'll actually be deployed, though. ChrisA From guido at python.org Sun Apr 6 00:08:38 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 5 Apr 2014 15:08:38 -0700 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: Lib2to3 has its own parser and tree so it can reconstruct the file, with comments. AST JUST THROWS AWAY TOO MUCH. On Saturday, April 5, 2014, Chris Angelico wrote: > On Sun, Apr 6, 2014 at 3:39 AM, Serhiy Storchaka > > wrote: > > What about strings? > > > >>>> print(ast.dump(ast.parse(r"x = '''\x0a\u000d''' r'x'"))) > > Module(body=[Assign(targets=[Name(id='x', ctx=Store())], > > value=Str(s='\n\rx'))]) > > > > Tuples? > > > >>>> print(ast.dump(ast.parse("x = (1, 2,)"))) > > Module(body=[Assign(targets=[Name(id='x', ctx=Store())], > > value=Tuple(elts=[Num(n=1), Num(n=2)], ctx=Load()))]) > > I don't have any concrete use cases for those, but the same could be > done for every type of node as a general mechanism for recreating a > file more exactly. Nick linked to a tracker issue with some discussion > on docstrings, which could benefit from that. Or maybe it would be > better to avoid the AST altogether and do a source-level translation, > as mentioned in the blog post he also linked to. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sun Apr 6 00:23:32 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 5 Apr 2014 16:23:32 -0600 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: On Apr 5, 2014 3:38 PM, "Nick Coghlan" wrote: > In the context of serialisation, one key lesson we have learned is that arbitrary ordering is a problem when you want to minimise spurious diffs, and sorting isn't a simple solution. Good point. > > Tools like doctest don't tolerate spurious diffs at all, but are often amenable to a sorting based answer. [snip] > These *can* all be done today, but *not* by using keyword arguments. In my view, the problem to be addressed is that keyword arguments *look* like they should work for these cases, because they have a definite order in the source code. The only reason they don't work is because the interpreter throws that ordering information away. > > It's a textbook case of a language feature becoming an attractive nuisance in some circumstances: the simple and obvious solution for the above use cases *doesn't actually work* for reasons that aren't obviously clear if you don't have a firm grasp of Python's admittedly complicated argument handling. That's a great way of putting it. > > The simplest way out that I can see? Just make it work, even if that means also preserving the order of arbitrary keyword arguments in cases that *dont* need it. So you're advocating simply using OrderedDict for kwargs? Ultimately that's my preference too. The other options are attempts at addressing concerns that have come up, particularly Guido's. Also, "just make it work" must include not having a meaningful impact on function call performance. :) That was the big objection to the original proposal(s) to just switch kwargs to OrderedDict. > > Deciding whether or not to provide a way to opt in to dropping the order info for speed and memory reasons then becomes a separate optimisation discussion *after* the current usability trap has been addressed. Sort of. The PEP certainly needs to address this point one way or the other. Unless Guido relaxes his objections, that is. :) Regardless, once I have a draft PEP out the next step will be to implement the OrderedDict swap and see how things look performance-wise. It may turn out that the impact is marginal, at which point the PEP won't have to say much. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Apr 6 02:08:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Apr 2014 10:08:56 +1000 Subject: [Python-ideas] Preserving **kwargs order In-Reply-To: References: <532A55AE.5040509@mrabarnett.plus.com> <1395297801.17430.YahooMailNeo@web181006.mail.ne1.yahoo.com> <532AE5F3.9040508@mrabarnett.plus.com> <87C2AE45-4FD2-418D-9288-B5BD51D06F27@yahoo.com> <3AB6F5EF-10B2-40AF-B557-33806B7573BA@yahoo.com> <1396515341.28255.YahooMailNeo@web181006.mail.ne1.yahoo.com> <1396576219.19686.YahooMailNeo@web181003.mail.ne1.yahoo.com> <1396619421.7911.102758557.3F31C249@webmail.messagingengine.com> Message-ID: On 6 Apr 2014 08:23, "Eric Snow" wrote: > > On Apr 5, 2014 3:38 PM, "Nick Coghlan" wrote: > > In the context of serialisation, one key lesson we have learned is that arbitrary ordering is a problem when you want to minimise spurious diffs, and sorting isn't a simple solution. > > Good point. > > > > > Tools like doctest don't tolerate spurious diffs at all, but are often amenable to a sorting based answer. > [snip] > > > These *can* all be done today, but *not* by using keyword arguments. In my view, the problem to be addressed is that keyword arguments *look* like they should work for these cases, because they have a definite order in the source code. The only reason they don't work is because the interpreter throws that ordering information away. > > > > It's a textbook case of a language feature becoming an attractive nuisance in some circumstances: the simple and obvious solution for the above use cases *doesn't actually work* for reasons that aren't obviously clear if you don't have a firm grasp of Python's admittedly complicated argument handling. > > That's a great way of putting it. > > > > > The simplest way out that I can see? Just make it work, even if that means also preserving the order of arbitrary keyword arguments in cases that *dont* need it. > > So you're advocating simply using OrderedDict for kwargs? Ultimately that's my preference too. The other options are attempts at addressing concerns that have come up, particularly Guido's. In my experience, the best way to go with a PEP is to make the best case you can for the solution you *really* want, and then let others make the case for why that isn't possible or acceptable. Adjusting the PEP accordingly as people persuade you to change your mind then helps ensure that the rationale for various design decisions is accurately recorded in the final PEP that is submitted for pronouncement. Acknowledge the known objections up front, and try to make a persuasive case to counter them. In this case, actually making the change and checking the impact on the macro and micro benchmark suites is likely the best way to bring actual data to bear on the question of the performance implications. The initial draft doesn't need to include that data, just a promise to collect it before submitting the PEP for a final decision (see Christian's hash PEP for an example of a "speed impact is likely real, but within acceptable limits given the other gains" proposal). > Also, "just make it work" must include not having a meaningful impact on function call performance. :) That was the big objection to the original proposal(s) to just switch kwargs to OrderedDict. Yep, but performance concerns are notorious for needing to be measured to see the true impact. A lot of speed critical code already uses positional args instead of keyword args, and APIs like str.format_map exist specifically to bypass the whole kwarg dance entirely. So it isn't *at all* clear how much overall impact this change will have outside micro benchmarks. It may also be possible to micro-optimise the "only 1 arbitrary kwarg" case to use an ordinary dict (if benchmarking backs up the stated concern that the simpler solution of always using odict may be too slow or use too much memory and handling cases like "key" arguments more efficiently then turns out to help address that). But we'll never know if we pre-emptively concede to the speculative concerns about the performance impact, rather than attempting to gather the data needed to show whether or not the simplest, most comprehensive solution can be attained with a negligible (or at least acceptable) speed impact. Cheers, Nick. > > > > > Deciding whether or not to provide a way to opt in to dropping the order info for speed and memory reasons then becomes a separate optimisation discussion *after* the current usability trap has been addressed. > > Sort of. The PEP certainly needs to address this point one way or the other. Unless Guido relaxes his objections, that is. :) Regardless, once I have a draft PEP out the next step will be to implement the OrderedDict swap and see how things look performance-wise. It may turn out that the impact is marginal, at which point the PEP won't have to say much. > > -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From atmb4u at gmail.com Sun Apr 6 04:58:24 2014 From: atmb4u at gmail.com (Anoop Thomas Mathew) Date: Sun, 6 Apr 2014 08:28:24 +0530 Subject: [Python-ideas] REG: Scope of a better API for JSON parser Message-ID: I went through the state of cpython JSON parser, and found that there is a scope for improvement in the parser interface. I have been thinking of implementing a class called QuickParse inside json module, which will tidy up the process of finding node, and *cuts down a number of for loops, if .. else and try ... except blocks.* For eg. sample_json = { 'name' : 'John Doe', 'age' : 45, 'level': 'Expert', 'languages': [ 'C', 'C++', 'Python', 'Clojure' ], 'account details': { 'account number': 1233312312, 'account balance': 1000000000.00 } } sample_json can be a file or string *json_doc = QuickParse(sample_json)* *json_doc.get(["name"])* 'John Doe' *json_doc.get(["languages", 2])* 'C++' *json_doc.get(["account details", "account balance"])* 1000000000.00 *json_doc.get(["notpresent"]) * None This is something I've been using in many number of projects, due to the complexity of the json documents these days. Also, there is a plan to provide option to use functions for iterating over the document, as well as selection ranges for expected values. Looking forward to hear everyone's views on this. regards, ATM -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Apr 6 07:26:06 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 5 Apr 2014 22:26:06 -0700 (PDT) Subject: [Python-ideas] REG: Scope of a better API for JSON parser In-Reply-To: References: Message-ID: <1396761966.10732.YahooMailNeo@web181006.mail.ne1.yahoo.com> From: Anoop Thomas Mathew Sent: Saturday, April 5, 2014 7:58 PM >I went through the state of cpython?JSON?parser, and found that there is a scope for improvement in the parser interface. >I have been thinking of implementing a class called QuickParse inside json module, which will tidy up the process of finding node, and?cuts down a number of for loops, if .. else and try ... except blocks. > >For eg. > >sample_json?= { >'name' : 'John Doe', >'age' : 45, >'level': 'Expert', >'languages': [ >? ? ? ? ? ? ? 'C', >? ? ? ? ? ? ??'C++', >? ? ? ? ? ? ??'Python', >? ? ? ? ? ? ??'Clojure' >? ? ? ? ? ? ??], >? ? 'account details': { >? ? ? ? ?'account number': 1233312312, >? ? ? ? ?'account balance': 1000000000.00 >? ? ?} >} That's not a JSON object, that's just a plain old Python dict. It doesn't even have anything to do with JSON (except the variable name, which is misleading). And you don't need to "parse" a dict; you just use [] or get on it. Even if you actually had a JSON string or file serializing this dict, you could get exactly the dict by just calling json.loads or json.load on it. It's hard to imagine anything simpler than that. Compare your desired code to the equivalents that already work today: >json_doc = QuickParse(sample_json) > >json_doc.get(["name"]) >'John Doe' sample_json['name'] ... or ... sample_json.get('name') Note that, besides the simple and more flexible access, you already didn't need that extra step of building a QuickParse object. >json_doc.get(["languages", 2]) >?'C++' sample_json['languages'][2] >json_doc.get(["account details", "account balance"]) >1000000000.00 sample_json['account details']['account balance'] >json_doc.get(["notpresent"])? >None sample_json.get('notpresent') >This is something I've been using in many number of projects, due to the complexity of the?json?documents these days. Also, there is a plan to provide option to use functions for iterating over the document, as well as selection ranges for expected values. You can already iterate over a dict, and again, I can't imagine how anything you design will be simpler than what you can already do. Also, at the start, you suggested that this would be an improvement to the JSON _parser_. But the stuff you showed has nothing to do with the parser. Unless the result of calling QuickParse is not JSON parsed into something dict-like, but rather something that holds the original JSON and re-parses it for each?"get" request? I'm going to assume that the whole bit about parsing is a red herring, and the part you actually want is the ability to call the "get" function with multiple keys?and recursively walk the structure. I'm sure you can find recipes for that on ActiveState or PyPI, but it's also very trivial to write yourself: ? ? def get(obj, *keys, default=None): ? ? ? ? for key in keys: ? ? ? ? ? ? try: ? ? ? ? ? ? ? ? obj = obj[key] ? ? ? ? ? ? except LookupError: ? ? ? ? ? ? ? ? return default ? ? ? ? return obj Now, without needing any kind of special object, you can write this: ? ? get(sample_json, 'languages', 2) From ericsnowcurrently at gmail.com Sun Apr 6 07:43:15 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 5 Apr 2014 23:43:15 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. Message-ID: Here's the proper proposal I promised. Hopefully it reflects our discussions and the past ones too. My goal here is to focus in on what I consider to be the most viable approach, which capturing the gist of the various concerns and alternatives. There are still a few small gaps that I will work on as time permits. Feedback is welcome. FYI, I also hope to have a basic implementation up in time for the language summit at PyCon (Wednesday) so that related discussions there might have something more concrete to support them. :) -eric ===================================================================== PEP: XXX Title: Preserving the order of \*\*kwargs in a function. Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow Discussions-To: python-ideas at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-Apr-2014 Python-Version: 3.5 Post-History: Resolution: Abstract ======== The \*\*kwargs syntax in a function definition indicates that the interpreter should collect all keyword arguments that do not correspond to other named parameters. However, Python does not preserved the order in which those collected keyword arguments were passed to the function. In some contexts the order matters. This PEP introduces a mechanism by which the passed order of collected keyword arguments will now be preserved. Motivation ========== Python's \*\*kwargs syntax in function definitions provides a powerful means of dynamically handling keyword arguments. In some applications of the syntax (see _`Use Cases`), the semantics applied to the collected keyword arguments requires that order be preserved. Unsurprisingly, this is similar to how OrderedDict is related to dict. Currently to preserved the order you have to do so manually and separately from the actual function call. This involves building an ordered mapping, whether an OrderedDict or an iterable of 2-tuples, which is then passed as a single argument to the function. [#arg_unpacking]_ With the capability described in this PEP, that boilerplate would no longer be required. For comparision, currently:: kwargs = OrderedDict() kwargs['eggs'] = ... ... def spam(a, kwargs): ... and with this proposal:: def spam(a, **kwargs): ... Nick Coglan, speaking of some of the uses cases, summed it up well [#nick_obvious]_:: These *can* all be done today, but *not* by using keyword arguments. In my view, the problem to be addressed is that keyword arguments *look* like they should work for these cases, because they have a definite order in the source code. The only reason they don't work is because the interpreter throws that ordering information away. It's a textbook case of a language feature becoming an attractive nuisance in some circumstances: the simple and obvious solution for the above use cases *doesn't actually work* for reasons that aren't obviously clear if you don't have a firm grasp of Python's admittedly complicated argument handling. This observation is supported by the appearance of this proposal over the years and the numerous times that people have been confused by the constructor for OrderedDict. [#past_threads]_ [#loss_of_order]_ [#compact_dict]_ Use Cases ========= As Nick noted, the current behavior of \*\*kwargs is unintuitive in cases where one would expect order to matter. Aside from more specific cases outlined below, in general "anything else where you want to control the iteration order *and* set field names and values in a single call will potentially benefit." [#nick_general]_ That matters in the case of factories (e.g. __init__()) for ordered types. Serialization ------------- Obviously OrderedDict would benefit (both __init__() and update()) from ordered kwargs. However, the benefit also extends to serialization APIs [#nick_obvious]_:: In the context of serialisation, one key lesson we have learned is that arbitrary ordering is a problem when you want to minimise spurious diffs, and sorting isn't a simple solution. Tools like doctest don't tolerate spurious diffs at all, but are often amenable to a sorting based answer. The cases where it would be highly desirable to be able use keyword arguments to control the order of display of a collection of key value pairs are ones like: * printing out key:value pairs in CLI output * mapping semantic names to column order in a CSV * serialising attributes and elements in particular orders in XML * serialising map keys in particular orders in human readable formats like JSON and YAML (particularly when they're going to be placed under source control) Debugging --------- In the words of Raymond Hettinger [#raymond_debug]_:: It makes it easier to debug if the arguments show-up in the order they were created. AFAICT, no purpose is served by scrambling them. Other Use Cases --------------- * Mock objects. [#mock]_ * Controlling object presentation. * Alternate namedtuple() where defaults can be specified. * Specifying argument priority by order. Concerns ======== Performance ----------- As already noted, the idea of ordered keyword arguments has come up on a number of occasions. Each time it has been met with the same response, namely that preserving keyword arg order would have a sufficiently adverse effect on function call performance that it's not worth doing. However, Guido noted the following [#guido_open]_:: Making **kwds ordered is still open, but requires careful design and implementation to avoid slowing down function calls that don't benefit. As will be noted below, there are ways to work around this at the expense of increased complication. Ultimately the simplest approach is the one that makes the most sense: pack collected key word arguments into an OrderedDict. However, without a C implementation of OrderedDict there isn't much to discuss. That should change in Python 3.5. [#c_ordereddict]_ In some cases the difference of performance between dict and OrderedDict *may* be of significance. For instance: when the collected kwargs has an extended lifetime outside the originating function or the number of collected kwargs is massive. However, the difference in performance (both CPU and memory) in those cases should not be significant. Furthermore, the performance of the C OrderedDict implementation is essentially identical with dict for the non-mutating API. A concrete representation of the difference in performance will be a part of this proposal before its resolution. Other Python Implementations ---------------------------- Another important issue to consider is that new features must be cognizant of the multiple Python implementations. At some point each of them would be expected to have implemented ordered kwargs. In this regard there doesn't seem to be an issue with the idea. [#ironpython]_ Each of the major Python implementations will be consulted regarding this proposal before its resolution. Specification ============= Starting in version 3.5 Python will preserve the order of keyword arguments as passed to a function. This will apply only to functions for which the definition uses the \*\*kwargs syntax for collecting otherwise unspecified keyword arguments. Only the order of those keyword arguments will be preserved. Relationship to **-unpacking syntax ----------------------------------- The ** unpacking syntax in function calls has no special connection with this proposal. Keyword arguments provided by unpacking will be treated in exactly the same way as they are now: ones that match defined parameters are gather there and the remainder will be collected into the ordered kwargs (just like any other unmatched keyword argument). Note that unpacking a mapping with undefined order, such as dict, will preserve its iteration order like normal. It's just that the order will remain undefined. The OrderedDict into which the unpacked key-value pairs will then be packed will not be able to provide any alternate ordering. This should not be surprising. There have been brief discussions of simply passing these mappings through to the functions kwargs without unpacking and repacking them, but that is both outside the scope of this proposal and probably a bad idea regardless. (There is a reason those discussions were brief.) Relationship to inspect.Signature --------------------------------- Signature objects should need no changes. The `kwargs` parameter of inspect.BoundArguments (returned by Signature.bind() and Signature.bind_partial()) will change from a dict to an OrderedDict. C-API ----- TBD Syntax ------ No syntax is added or changed by this proposal. Backward-Compatibility ---------------------- The following will change: * type(kwargs) * iteration order of kwargs will now be consistent (except of course in the case described above) * as already noted, performance will be marginally different None of these should be an issue. However, each will be carefully considered while this proposal is under discussion. Reference Implementation ======================== TBD Implementation Notes -------------------- TBD Alternate Approaches ==================== Opt-out Decorator ----------------- This is identical to the current proposal with the exception that Python would also provide a decorator in functools that would cause collected keyword arguments to be packed into a normal dict instead of an OrderedDict. Prognosis: This would only be necessary if performance is determined to be significantly different in some uncommon cases or that there are other backward-compatibility concerns that cannot be resolved otherwise. Opt-in Decorator ---------------- The status quo would be unchanged. Instead Python would provide a decorator in functools that would register or mark the decorated function as one that should get ordered keyword arguments. The performance overhead to check the function at call time would be marginal. Prognosis: The only real down-side is in the case of function wrappers factories (e.g. functools.partial and many decorators) that aim to perfectly preserve keyword arguments by using kwargs in the wrapper definition and kwargs unpacking in the call to the wrapped function. Each wrapper would have to be updated separately, though having functools.wraps() do this automaticallywould help. __kworder__ ----------- The order of keyword arguments would be stored separately in a list at call time. The list would be bound to __kworder__ in the function locals. Prognosis: This likewise complicates the wrapper case. Compact dict with faster iteration ---------------------------------- Raymond Hettinger has introduced the idea of a dict implementation that would result in preserving insertion order on dicts (until the first deletion). This would be a perfect fit for kwargs. [#compact_dict]_ Prognosis: The idea is still uncertain in both viability and timeframe. ***kwargs --------- This would add a new form to a function's signature as a mutually exclusive parallel to \*\*kwargs. The new syntax, ***kwargs (note that there are three asterisks), would indicate that kwargs should preserve the order of keyword arguments. Prognosis: New syntax is only added to Python under the most *dire* circumstances. With other available solutions, new syntax is not justifiable. Furthermore, like all opt-in solutions, the new syntax would complicate the pass-through case. annotations ----------- This is a variation on the decorator approach. Instead of using a decorator to mark the function, you would use a function annotation on \*\*kwargs. Prognosis: In addition to the pass-through complication, annotations have been actively discouraged in Python core development. Use of annotations to opt-in to order preservation runs the risk of interfering with other application-level use of annotations. dict.__order__ -------------- dict objects would have a new attribute, `__order__` that would default to None and that in the kwargs case the interpreter would use in the same way as described above for __kworder__. Prognosis: It would mean zero impact on kwargs performance but the change would be pretty intrusive (Python uses dict a lot). Also, for the wrapper case the interpreter would have to be careful to preserve `__order__`. KWArgsDict.__order__ -------------------- This is the same as the `dict.__order__` idea, but kwargs would be an instance of a new minimal dict subclass that provides the `__order__` attribute. dict would instead be unchanged. Prognosis: Simply switching to OrderedDict is a less complicated and more intuitive change. Acknowledgements ================ Thanks to Andrew Barnert for helpful feedback and to the participants of all the past email threads. Footnotes ========= .. [#arg_unpacking] Alternately, you could also replace ** in your function definition with * and then pass in key/value 2-tuples. This has the advantage of not requiring the keys to be valid identifier strings. See https://mail.python.org/pipermail/python-ideas/2014-April/027491.html. References ========== .. [#nick_obvious] https://mail.python.org/pipermail/python-ideas/2014-April/027512.html .. [#past_threads] https://mail.python.org/pipermail/python-ideas/2009-April/004163.html https://mail.python.org/pipermail/python-ideas/2010-October/008445.html https://mail.python.org/pipermail/python-ideas/2011-January/009037.html https://mail.python.org/pipermail/python-ideas/2013-February/019690.html https://mail.python.org/pipermail/python-ideas/2013-May/020727.html https://mail.python.org/pipermail/python-ideas/2014-March/027225.html http://bugs.python.org/issue16276 http://bugs.python.org/issue16553 http://bugs.python.org/issue19026 http://bugs.python.org/issue5397#msg82972 .. [#loss_of_order] https://mail.python.org/pipermail/python-dev/2007-February/071310.html .. [#compact_dict] https://mail.python.org/pipermail/python-dev/2012-December/123028.html https://mail.python.org/pipermail/python-dev/2012-December/123105.html https://mail.python.org/pipermail/python-dev/2013-May/126327.html https://mail.python.org/pipermail/python-dev/2013-May/126328.html .. [#nick_general] https://mail.python.org/pipermail/python-dev/2012-December/123105.html .. [#raymond_debug] https://mail.python.org/pipermail/python-dev/2013-May/126327.html .. [#mock] https://mail.python.org/pipermail/python-ideas/2009-April/004163.html https://mail.python.org/pipermail/python-ideas/2009-April/004165.html https://mail.python.org/pipermail/python-ideas/2009-April/004175.html .. [guido_open] https://mail.python.org/pipermail/python-dev/2013-May/126404.html .. [#c_ordereddict] http://bugs.python.org/issue16991 .. [#ironpython] https://mail.python.org/pipermail/python-dev/2012-December/123100.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From abarnert at yahoo.com Sun Apr 6 10:18:37 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 6 Apr 2014 01:18:37 -0700 (PDT) Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: <1396772317.78213.YahooMailNeo@web181001.mail.ne1.yahoo.com> I like it, but a couple?comments: From: Eric Snow Sent: Saturday, April 5, 2014 10:43 PM > Starting in version 3.5 Python will preserve the order of keyword > arguments as passed to a function.? This will apply only to functions > for which the definition uses the \*\*kwargs syntax for collecting > otherwise unspecified keyword arguments.? Only the order of those > keyword arguments will be preserved. Will this be an OrderedDict, or just some mapping that (unless later modified?)?iterates in the order of the keywords as passed? The latter might be nice, because it allows implementations to do things like use Raymond Hettinger's special dict, or a wrapper around a native mapping, or something else that isn't appropriate for an OrderedDict implementation but is good enough for this purpose. But I'm not sure how you'd word that in the language reference; the two relevant sentences are already pretty long and complex today (in sections 6.3.4 and 8.6): >?If any keyword argument does not correspond to a formal parameter name, a TypeError exception is raised, unless a formal parameter using the syntax **identifier is present; in this case, that formal parameter receives a dictionary containing the excess keyword arguments (using the keywords as keys and the argument values as corresponding values), or a (new) empty dictionary if there were no excess keyword arguments. >?If the form ?**identifier? is present, it is initialized to a new dictionary receiving any excess keyword arguments, defaulting to a new empty dictionary. Parameters after ?*? or ?*identifier? are keyword-only parameters and may only be passed used keyword arguments. Also: > Relationship to **-unpacking syntax > ----------------------------------- > > The ** unpacking syntax in function calls has no special connection with > this proposal.? Keyword arguments provided by unpacking will be treated > in exactly the same way as they are now: ones that match defined > parameters are gather there and the remainder will be collected into the > ordered kwargs (just like any other unmatched keyword argument). I think you want to explicitly specify that they will be processed after any normal keyword arguments, and in the mapping's iteration order. (Otherwise, partial, perfect-forwarding wrappers, etc. would have obvious problems.)?That isn't specified by Python 3.4; the language reference (6.3.4 again, a few paragraphs down) says: >?If the syntax **expression appears in the function call, expression must evaluate to a mapping, the contents of which are treated as additional keyword arguments. CPython 3.4, if you just changed it to use an OrderedDict instead of a dict, would?put the expression's contents _before_ the normal keywords.?And there are other reasonable ways to implement things that would be perfectly valid under the current language definition that might end up with the mapping's contents in reverse order, or even arbitrary order. If you want to force those implementations to change (which you obviously do), I think the language reference should reflect that. Finally: > Opt-out Decorator > ----------------- > > This is identical to the current proposal with the exception that Python > would also provide a decorator in functools that would cause collected > keyword arguments to be packed into a normal dict instead of an > OrderedDict. I think you may need a bit more of an argument against this, because there are really two issues here. First, there's the everyday case: every function that takes **kwargs will get a little slower. The optimized C OrderedDict will hopefully make this not significant enough to worry about; if not, the opt-out decorator may be necessary. Second, there's Guido's case: functions that keep kwargs around for later will be potentially confusing, conceptually wrong, and possibly lead to significantly less speed- or memory-efficient code, especially if they later add a whole bunch of other stuff to the stored kwargs dict.?The C OrderedDict isn't going to help here. The opt-out decorator would, but just storing dict(kwargs) instead of kwargs solves it more explicitly, just as simply, and with no more changes to existing code. So, the only reason the opt-out decorator would be necessary for this case is if the cost of that dict(kwargs) is too high. Which seems unlikely for most realistic cases?the performance issue is about adding thousands of items later to an initially-small kwargs dict, not about receiving thousands of keywords, right? On the other hand, I understand Nick's point about not trying to answer arguments that people may never make in your PEP, so maybe it's better to leave this part as-is. From atmb4u at gmail.com Sun Apr 6 11:55:53 2014 From: atmb4u at gmail.com (Anoop Thomas Mathew) Date: Sun, 6 Apr 2014 15:25:53 +0530 Subject: [Python-ideas] REG: Scope of a better API for JSON parser In-Reply-To: <1396761966.10732.YahooMailNeo@web181006.mail.ne1.yahoo.com> References: <1396761966.10732.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: On 6 April 2014 10:56, Andrew Barnert wrote: > From: Anoop Thomas Mathew > Sent: Saturday, April 5, 2014 7:58 PM > > > >I went through the state of cpython JSON parser, and found that there is > a scope for improvement in the parser interface. > > > >I have been thinking of implementing a class called QuickParse inside > json module, which will tidy up the process of finding node, and cuts down > a number of for loops, if .. else and try ... except blocks. > > > >For eg. > > > >sample_json = { > >'name' : 'John Doe', > >'age' : 45, > >'level': 'Expert', > >'languages': [ > > 'C', > > 'C++', > > 'Python', > > 'Clojure' > > ], > > 'account details': { > > 'account number': 1233312312, > > 'account balance': 1000000000.00 > > } > >} > > That's not a JSON object, that's just a plain old Python dict. It doesn't > even have anything to do with JSON (except the variable name, which is > misleading). > > And you don't need to "parse" a dict; you just use [] or get on it. > > Even if you actually had a JSON string or file serializing this dict, you > could get exactly the dict by just calling json.loads or json.load on it. > It's hard to imagine anything simpler than that. > > Compare your desired code to the equivalents that already work today: > > >json_doc = QuickParse(sample_json) > > > >json_doc.get(["name"]) > >'John Doe' > > sample_json['name'] > ... or ... > sample_json.get('name') > > Note that, besides the simple and more flexible access, you already didn't > need that extra step of building a QuickParse object. > > >json_doc.get(["languages", 2]) > > > 'C++' > > > sample_json['languages'][2] > > >json_doc.get(["account details", "account balance"]) > >1000000000.00 > > > sample_json['account details']['account balance'] > > >json_doc.get(["notpresent"]) > >None > > > sample_json.get('notpresent') > I completely agree to your point. But what if the document is of something like this: person['country']['state']['city']['county'] > >This is something I've been using in many number of projects, due to the > complexity of the json documents these days. Also, there is a plan to > provide option to use functions for iterating over the document, as well as > selection ranges for expected values. > > > You can already iterate over a dict, and again, I can't imagine how > anything you design will be simpler than what you can already do. > > Also, at the start, you suggested that this would be an improvement to the > JSON _parser_. But the stuff you showed has nothing to do with the parser. > Unless the result of calling QuickParse is not JSON parsed into something > dict-like, but rather something that holds the original JSON and re-parses > it for each "get" request? > > I'm going to assume that the whole bit about parsing is a red herring, and > the part you actually want is the ability to call the "get" function with > multiple keys and recursively walk the structure. I'm sure you can find > recipes for that on ActiveState or PyPI, but it's also very trivial to > write yourself: > > > def get(obj, *keys, default=None): > for key in keys: > try: > obj = obj[key] > except LookupError: > return default > return obj I apologize for making the previous mail complicated. You have pinned down what I was suggesting. My suggestion is to integrate something similar as you suggested above to the cpython json module. Won't that be a good feature for developers to be able to write code without worrying if it going to fail and writing may try.. except or if else to handle those edge cases? I definitely feel that it would be a great feature which is common for any program which is parsing multilevel json documents. > Now, without needing any kind of special object, you can write this: > > get(sample_json, 'languages', 2) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Apr 6 12:33:48 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 6 Apr 2014 03:33:48 -0700 (PDT) Subject: [Python-ideas] REG: Scope of a better API for JSON parser In-Reply-To: References: <1396761966.10732.YahooMailNeo@web181006.mail.ne1.yahoo.com> Message-ID: <1396780428.1308.YahooMailNeo@web181001.mail.ne1.yahoo.com> From: Anoop Thomas Mathew Sent: Sunday, April 6, 2014 2:55 AM >On 6 April 2014 10:56, Andrew Barnert wrote: > >From: Anoop Thomas Mathew [snip] >>>json_doc.get(["account details", "account balance"]) >>>1000000000.00 >> >>sample_json['account details']['account balance'] >> >I completely agree to your point. But what if the document is of something like this: > >person['country']['state']['city']['county'] So what? Do you not understand that? It's simple, explicit, readable, and identical to code you could write in other languages like JavaScript, Ruby, etc. This is a familiar, easily-recognizable pattern from code dealing with complex heterogeneous objects in all of these languages. Do you really want to do away with that for the sake of saving a few keystrokes? [snip] >>I'm going to assume that the whole bit about parsing is a red herring, and the part you actually want is the ability to call the "get" function with multiple keys?and recursively walk the structure. I'm sure you can find recipes for that on ActiveState or PyPI, but it's also very trivial to write yourself: >>? >>? ? def get(obj, *keys, default=None): >>? ? ? ? for key in keys: >>? ? ? ? ? ? try: >>? ? ? ? ? ? ? ? obj = obj[key] >>? ? ? ? ? ? except LookupError: >>? ? ? ? ? ? ? ? return default >>? ? ? ? return obj > >I apologize for making the previous mail complicated. You have pinned down what I was suggesting. > >My suggestion is to integrate something similar as you suggested above to the cpython json module. Again: Why would you put it in the json module? You don't have any JSON anything anywhere; you have a dict with dicts inside it.?If you want a simple function that works on any indexable object full of indexable objects, that function will work the same way whether you give it something you parsed from a JSON string you pulled off the wire, something you built up yourself out of database queries,?a multi-dimensional array, a tree, etc. >Won't that be a good feature for developers to be able to write code without worrying if it going to fail and writing may try.. except or if else to handle those edge cases?? No, because Python already has features for that. First, you don't need many try/excepts, just a single one, no matter how many possibly-raising lookups you're doing: ? ? >>> try: ? ? ... ? ? val = d['stuff']['things'][7] ? ? ... except LookupError: ? ? ... ? ? val = None It doesn't matter whether d['stuff'] was missing, or d['stuff']['things'], or d['stuff']['things'][7]; they'll all be handled by a single try. And if you don't want to worry about try/except (or if/else, but you really shouldn't ever be using if/else for cases like this), dict already has a get method: ? ? >>> d = {'a': 1, 'b': 2} ? ? >>> d['c'] ? ? KeyError: 'c' ? ? >>> d.get('c') ? ? None ? ? >>> d.get('c', 0) ? ? 0 Also, different projects are likely to need different variations on the same theme. Sometimes you want a Cocoa-KV-style dotted string of keys, or a Windows-registry-style slashed string, instead of an iterable. Sometimes you only want to handle recursive mappings or recursive sequences, sometimes recursive mappings-and-sequences. Sometimes you want to return a default value other than None, maybe even one that's dependent on where the lookup failed. Sometimes you want to look up multiple key-paths at once ala operator.itemgetter. And so on. All of these are dead-simple to write yourself, so any project that needs some version of this should just include the version that it needs. From ncoghlan at gmail.com Sun Apr 6 12:39:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 Apr 2014 20:39:51 +1000 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: On 6 April 2014 15:43, Eric Snow wrote: > Here's the proper proposal I promised. Hopefully it reflects our > discussions and the past ones too. My goal here is to focus in on > what I consider to be the most viable approach, which capturing the > gist of the various concerns and alternatives. There are still a few > small gaps that I will work on as time permits. Feedback is welcome. Nice! One specific comment below. > Relationship to **-unpacking syntax > ----------------------------------- > > The ** unpacking syntax in function calls has no special connection with > this proposal. Keyword arguments provided by unpacking will be treated > in exactly the same way as they are now: ones that match defined > parameters are gather there and the remainder will be collected into the > ordered kwargs (just like any other unmatched keyword argument). There *is* a connection here: this needs to play nice with the ordered kwargs in order to handle the pass-through case correctly. The guarantee that the order of the supplied mapping (minus any entries that map to named parameters) will be preserved needs to be made explicit. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ericsnowcurrently at gmail.com Sun Apr 6 18:34:39 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 6 Apr 2014 10:34:39 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: On Apr 6, 2014 4:39 AM, "Nick Coghlan" wrote: > There *is* a connection here: this needs to play nice with the ordered > kwargs in order to handle the pass-through case correctly. > > The guarantee that the order of the supplied mapping (minus any > entries that map to named parameters) will be preserved needs to be > made explicit. You're right. It's already implied by OrderedDict. However the point *should* be explicit. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Apr 6 18:20:48 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 06 Apr 2014 09:20:48 -0700 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: References: Message-ID: <53417EE0.9050306@stoneleaf.us> On 04/05/2014 03:08 PM, Guido van Rossum wrote: > > Lib2to3 has its own parser and tree so it can reconstruct the file, > with comments. AST just throws away too much. Are you saying it throws away too much for the question at hand, or that it throws away too much and should be modified to not throw away too much? -- ~Ethan~ From guido at python.org Sun Apr 6 18:49:41 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Apr 2014 09:49:41 -0700 Subject: [Python-ideas] Retain string form of AST Numbers In-Reply-To: <53417EE0.9050306@stoneleaf.us> References: <53417EE0.9050306@stoneleaf.us> Message-ID: The former. It's porpose is to generate code, not to reproduce the source. On Sunday, April 6, 2014, Ethan Furman wrote: > On 04/05/2014 03:08 PM, Guido van Rossum wrote: > >> >> Lib2to3 has its own parser and tree so it can reconstruct the file, >> with comments. AST just throws away too much. >> > > Are you saying it throws away too much for the question at hand, or that > it throws away too much and should be modified to not throw away too much? > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skreft at gmail.com Sun Apr 6 19:32:05 2014 From: skreft at gmail.com (Sebastian Kreft) Date: Sun, 6 Apr 2014 14:32:05 -0300 Subject: [Python-ideas] [RFC] Draft-PEP: Adding structured data to built-in exceptions Message-ID: Continuing the discussion started in https://mail.python.org/pipermail/python-ideas/2014-February/025601.html, attached is a draft PEP explaining the rationale of the idea and the required changes. I'm looking forward to your comments. -- Sebastian Kreft -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- PEP: XXX Title: Sample reStructuredText PEP Template Version: $Revision$ Last-Modified: $Date$ Author: Sebastian Kreft Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Mar-2014 Post-History: Abstract ======== Exceptions like ``KeyError``, ``IndexError``, ``AttributeError`` and ``NameError`` do not provide all information required by programmers to debug and better understand what caused them. Furthermore, in some cases the messages even have slightly different formats, which makes it really difficult for tools to automatically provide additional information to diagnose the problem. To tackle the former and to lay ground for the latter, it is proposed to expand these exceptions so to hold both the offending and affected entities. Rationale ========= The main issue this PEP aims to solve is the fact that currently error messages are not that expressive and lack some key information to resolve the exceptions. Additionally, the information present on the error message is not always in the same format, which makes it very difficult for third-party libraries to provide automated diagnosis of the error. These automated tools could, for example, detect typos or display or log extra debug information. These could be particularly useful when running tests or in a long running application. Although it is in theory possible to have such libraries, they need to resort to hacks in order to achieve the goal. One such example is python-improved-exceptions [1]_, which modifies the byte-code to keep references to the possibly interesting objects and also parses the error messages to extract information like types or names. Unfortunately, such approach is extremely fragile and not portable. A similar proposal [2]_ has been implemented for ``ImportError`` and in the same fashion this idea has received support [3]_. Examples ======== IndexError ---------- The error message does not reference the list's length nor the index used. :: a = [1, 2, 3, 4, 5] a[5] IndexError: list index out of range KeyError -------- By convention the key is the first element of the error's argument, but there's no other information regarding the affected dictionary (keys types, size, etc.) :: b = {'foo': 1} b['fo'] KeyError: 'fo' AttributeError -------------- The object's type and the offending attribute are part of the error message. However, there are some different formats and the information is not always available. Furthermore, although the object type is useful in some cases, given the dynamic nature of Python, it would be much more useful to have a reference to the object itself. Additionally the reference to the type is not fully qualified and in some cases the type is just too generic to provide useful information, for example in case of accessing a module's attribute. :: c = object() c.foo AttributeError: 'object' object has no attribute 'foo' import string string.foo AttributeError: 'module' object has no attribute 'foo' a = string.Formatter() a.foo AttributeError: 'Formatter' object has no attribute 'foo' NameError --------- The error message provides typically the name. :: foo = 1 fo NameError: global name 'fo' is not defined Other Cases ----------- Issues are even harder to debug when the target object is the result of another expression, for example: :: a[b[c[0]]] This issue is also related to the fact that opcodes only have line number information and not the offset. This proposal would help in this case but not as much as having offsets. Proposal ======== Extend the exceptions ``IndexError``, ``KeyError``, ``AttributeError`` and ``NameError`` with the following: * ``IndexError``: target, index * ``KeyError``: target, key * ``AttributeError``: target, attribute * ``NameError``: name, scope? To remain backwards compatible these new attributes will be optional and keyword only. It is proposed to add this information, rather than just improve the error, as the former would allow new debugging frameworks and tools and also in the future to switch to a lazy generated message. Generated messages are discussed in [2]_, although they are not implemented at the moment. They would not only save some resources, but also uniform the messages. Potential Uses ============== An automated tool could for example search for similar keys within the object, allowing to display the following::: a = {'foo': 1} a['fo'] KeyError: 'fo'. Did you mean 'foo'? foo = 1 fo NameError: global name 'fo' is not defined. Did you mean 'foo'? See [3]_ for the output a TestRunner could display. Performance =========== Filling these new attributes would only require two extra parameters with data already available so the impact should be marginal. However, it may need special care for ``KeyError`` as the following pattern is already widespread. :: try: a[foo] = a[foo] + 1 except: a[foo] = 0 Note as well that storing these objects into the error itself would allow the lazy generation of the error message, as discussed in [2]_. References ========== .. [1] Python Exceptions Improved (https://www.github.com/sk-/python-exceptions-improved) .. [2] ImportError needs attributes for module and file name (http://bugs.python.org/issue1559549) .. [3] Enhance exceptions by attaching some more information to them (https://mail.python.org/pipermail/python-ideas/2014-February/025601.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Apr 6 19:37:30 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Apr 2014 10:37:30 -0700 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: On Sun, Apr 6, 2014 at 9:34 AM, Eric Snow wrote: > > On Apr 6, 2014 4:39 AM, "Nick Coghlan" wrote: > > There *is* a connection here: this needs to play nice with the ordered > > kwargs in order to handle the pass-through case correctly. > > > > The guarantee that the order of the supplied mapping (minus any > > entries that map to named parameters) will be preserved needs to be > > made explicit. > > You're right. It's already implied by OrderedDict. However the point > *should* be explicit. > Somehow the specification section in the PEP doesn't explicitly say it, but I presume that the type of kwargs will change from dict to OrderedDict? That seems simple enough, but the proof is in the pudding. How simple was it to implement? How did the speed compare? Until we have those answers we shouldn't accept this PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sun Apr 6 21:02:55 2014 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 6 Apr 2014 22:02:55 +0300 Subject: [Python-ideas] Relax __exit__ method to be a generator Message-ID: Literally it may be generator itself or function that returns generator object. Now I'm working on postgres library for asyncio (http://aiopg.readthedocs.org/). And I would to use *with statement* for transaction handling like: with (yield from cursor.transaction()): yield from cursor.execute(sql) The problem is: at exit of *with statement* I need to call `yield from cursor.execute('COMMIT')` or `yield from cursor.execute('ROLLBACK')`. I can do it only in __exit__ in *context manager*, but python understand only if __exit__: - returns true value, that suppresses exception from code block - returns None or any false value to propagate exception if any - raises exception itself I propose to add new rule: IF the code object is generator (co_flags & CO_GENERATOR) and __exit__ returns generator object (isinstance(ret, types.GeneratorType)) THEN do `yield from ret`. That's work fine if __exit__ itself is a *generator function* (contains `yield` or `yield from` statements): call to *generator function* returns *generator object*. The proposal: 1. Doesn't break any existing code except if user accidentally returns generator object instead of True from __exit__ call (he should not to do this and I sure this is very rare case). 2. Don't requires new syntax. asyncio itself uses: with (yield from lock): BLOCK for locking etc but unlocking for asyncio objects doesn't requires any `yield from`, so __exit__ code is just plain function but not generator. Also I can live with asyncio trick for __enter__: https://code.google.com/p/tulip/source/browse/asyncio/locks.py#156 The way is a but annoying but unrolling a value returned by __enter__ if the value is generator object will break existing code, sure. Thoughts? -- Thanks, Andrew Svetlov From ericsnowcurrently at gmail.com Sun Apr 6 21:06:29 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 6 Apr 2014 13:06:29 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: On Apr 6, 2014 11:37 AM, "Guido van Rossum" wrote: > Somehow the specification section in the PEP doesn't explicitly say it, but I presume that the type of kwargs will change from dict to OrderedDict? That seems simple enough, but the proof is in the pudding. How simple was it to implement? How did the speed compare? Until we have those answers we shouldn't accept this PEP. It's just a minor detail. You are correct, of course. I'll fix the PEP. I'll also post a performance comparison here once I have something in the next few days. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Apr 7 00:02:00 2014 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 6 Apr 2014 23:02:00 +0100 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <5322EC7D.1020908@egenix.com> <53230E87.7070503@stoneleaf.us> <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> Message-ID: On Sun, Mar 16, 2014 at 1:05 PM, Nick Coghlan wrote: > On 16 March 2014 22:26, Antoine Pitrou wrote: >> On Sun, 16 Mar 2014 12:57:06 +1000 >> Nick Coghlan wrote: >>> >>> Guido has asked them to have that discussion and summarise their >>> conclusions in the PEP - we can lob our armchair opinions into the mix >>> after the experts have spoken. >> >> Debunking armchair opinions is one of the points of the PEP process :-) > > Aye, but at the moment it makes sense to wait and see if there is even > an argument to be had - Nathaniel may decide that even after reviewing > the question seriously, he doesn't want to propose a right associative > operator :) And indeed, that does seem to be the way things have worked out :-). http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069834.html -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From guido at python.org Mon Apr 7 01:12:51 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Apr 2014 16:12:51 -0700 Subject: [Python-ideas] Relax __exit__ method to be a generator In-Reply-To: References: Message-ID: I prefer to use a try/finally statement. One of the points of using yield-from is that you can always tell where your code may be suspended by searching for "yield from". With your proposed change that would no longer be true -- any with statement would also have to be inspected, and there would no longer be a way to know from the source alone whether it might yield or not (because it would dynamic -- there's no way to be sure at compile time which context manager is being used). On Sun, Apr 6, 2014 at 12:02 PM, Andrew Svetlov wrote: > Literally it may be generator itself or function that returns generator > object. > > Now I'm working on postgres library for asyncio ( > http://aiopg.readthedocs.org/). > > And I would to use *with statement* for transaction handling like: > > with (yield from cursor.transaction()): > yield from cursor.execute(sql) > > The problem is: at exit of *with statement* I need to call `yield from > cursor.execute('COMMIT')` or `yield from cursor.execute('ROLLBACK')`. > > I can do it only in __exit__ in *context manager*, but python > understand only if __exit__: > - returns true value, that suppresses exception from code block > - returns None or any false value to propagate exception if any > - raises exception itself > > I propose to add new rule: > IF the code object is generator (co_flags & CO_GENERATOR) and __exit__ > returns generator object (isinstance(ret, types.GeneratorType)) > THEN do `yield from ret`. > > That's work fine if __exit__ itself is a *generator function* > (contains `yield` or `yield from` statements): call to *generator > function* returns *generator object*. > > The proposal: > 1. Doesn't break any existing code except if user accidentally > returns generator object instead of True from __exit__ call (he should > not to do this and I sure this is very rare case). > 2. Don't requires new syntax. > > asyncio itself uses: > > with (yield from lock): > BLOCK > > for locking etc but unlocking for asyncio objects doesn't requires any > `yield from`, so __exit__ code is just plain function but not > generator. > > Also I can live with asyncio trick for __enter__: > https://code.google.com/p/tulip/source/browse/asyncio/locks.py#156 > The way is a but annoying but unrolling a value returned by __enter__ > if the value is generator object will break existing code, sure. > > Thoughts? > > > -- > Thanks, > Andrew Svetlov > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Apr 7 01:20:38 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 7 Apr 2014 09:20:38 +1000 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> Message-ID: <20140406232038.GG16466@ando> On Sun, Apr 06, 2014 at 11:02:00PM +0100, Nathaniel Smith wrote: > On Sun, Mar 16, 2014 at 1:05 PM, Nick Coghlan wrote: > > Aye, but at the moment it makes sense to wait and see if there is even > > an argument to be had - Nathaniel may decide that even after reviewing > > the question seriously, he doesn't want to propose a right associative > > operator :) > > And indeed, that does seem to be the way things have worked out :-). > http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069834.html That's a shame in one way -- for anyone using operator overloading to define their own DSL, there's only ** if you want a right-associative operator to overload. Still, that's an extremely marginal, and hypothetical, use-case. Left-associative it is. Was that the last blocker for the PEP? -- Steven From guido at python.org Mon Apr 7 01:14:11 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Apr 2014 16:14:11 -0700 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: Also, I recommend you go ahead and commit it to the peps repo. Pick the next available number (looks like 468). Be sure that "make pep-0468.html pep-0000.html" doesn't emit any errors. On Sun, Apr 6, 2014 at 12:06 PM, Eric Snow wrote: > > On Apr 6, 2014 11:37 AM, "Guido van Rossum" wrote: > > Somehow the specification section in the PEP doesn't explicitly say it, > but I presume that the type of kwargs will change from dict to OrderedDict? > That seems simple enough, but the proof is in the pudding. How simple was > it to implement? How did the speed compare? Until we have those answers we > shouldn't accept this PEP. > > It's just a minor detail. You are correct, of course. I'll fix > the PEP. I'll also post a performance comparison here once I have > something in the next few days. > > -eric > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Mon Apr 7 01:44:27 2014 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Mon, 7 Apr 2014 02:44:27 +0300 Subject: [Python-ideas] Relax __exit__ method to be a generator In-Reply-To: References: Message-ID: Well, this is good point.. But from my experience try/finally is a common point of errors. Beginners use "with" properly as: with open(filename) as f: f.read() but switch to try/finally produces very common error. Instead of f = open(filename) try: f.read() finally: f.close() people usually write try: f = open(filename) f.read() finally: f.close() I saw it constantly many times. When I wrote an article about the problem and true solution in my blog I got several emails from my readers: please, tell me again why I need to move open() out of try block if it can generate exception on file opening? So maybe you would like some other syntax? For example with from obj: BLOCK That form can use magic methods with names different than __enter__/__exit__ (and get rid of "with (yield from lock)" BTW). And it's obviously? points on two suspending points: at BLOCK enter and exit. On Mon, Apr 7, 2014 at 2:12 AM, Guido van Rossum wrote: > I prefer to use a try/finally statement. One of the points of using > yield-from is that you can always tell where your code may be suspended by > searching for "yield from". With your proposed change that would no longer > be true -- any with statement would also have to be inspected, and there > would no longer be a way to know from the source alone whether it might > yield or not (because it would dynamic -- there's no way to be sure at > compile time which context manager is being used). > > > On Sun, Apr 6, 2014 at 12:02 PM, Andrew Svetlov > wrote: >> >> Literally it may be generator itself or function that returns generator >> object. >> >> Now I'm working on postgres library for asyncio >> (http://aiopg.readthedocs.org/). >> >> And I would to use *with statement* for transaction handling like: >> >> with (yield from cursor.transaction()): >> yield from cursor.execute(sql) >> >> The problem is: at exit of *with statement* I need to call `yield from >> cursor.execute('COMMIT')` or `yield from cursor.execute('ROLLBACK')`. >> >> I can do it only in __exit__ in *context manager*, but python >> understand only if __exit__: >> - returns true value, that suppresses exception from code block >> - returns None or any false value to propagate exception if any >> - raises exception itself >> >> I propose to add new rule: >> IF the code object is generator (co_flags & CO_GENERATOR) and __exit__ >> returns generator object (isinstance(ret, types.GeneratorType)) >> THEN do `yield from ret`. >> >> That's work fine if __exit__ itself is a *generator function* >> (contains `yield` or `yield from` statements): call to *generator >> function* returns *generator object*. >> >> The proposal: >> 1. Doesn't break any existing code except if user accidentally >> returns generator object instead of True from __exit__ call (he should >> not to do this and I sure this is very rare case). >> 2. Don't requires new syntax. >> >> asyncio itself uses: >> >> with (yield from lock): >> BLOCK >> >> for locking etc but unlocking for asyncio objects doesn't requires any >> `yield from`, so __exit__ code is just plain function but not >> generator. >> >> Also I can live with asyncio trick for __enter__: >> https://code.google.com/p/tulip/source/browse/asyncio/locks.py#156 >> The way is a but annoying but unrolling a value returned by __enter__ >> if the value is generator object will break existing code, sure. >> >> Thoughts? >> >> >> -- >> Thanks, >> Andrew Svetlov >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) -- Thanks, Andrew Svetlov From ericsnowcurrently at gmail.com Mon Apr 7 03:00:15 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 6 Apr 2014 19:00:15 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: On Sun, Apr 6, 2014 at 5:14 PM, Guido van Rossum wrote: > Also, I recommend you go ahead and commit it to the peps repo. Pick the next > available number (looks like 468). Be sure that "make pep-0468.html > pep-0000.html" doesn't emit any errors. Updated and committed. PEP 468 it is. Thanks. -eric From guido at python.org Mon Apr 7 03:04:55 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 6 Apr 2014 18:04:55 -0700 Subject: [Python-ideas] Relax __exit__ method to be a generator In-Reply-To: References: Message-ID: Well, for new syntax, the bar is pretty high... And we should probably move to python-ideas. On Sun, Apr 6, 2014 at 4:44 PM, Andrew Svetlov wrote: > Well, this is good point.. > > But from my experience try/finally is a common point of errors. > Beginners use "with" properly as: > > with open(filename) as f: > f.read() > > but switch to try/finally produces very common error. > > Instead of > > f = open(filename) > try: > f.read() > finally: > f.close() > > people usually write > > try: > f = open(filename) > f.read() > finally: > f.close() > > I saw it constantly many times. When I wrote an article about the > problem and true solution in my blog I got several emails from my > readers: please, tell me again why I need to move open() out of try > block if it can generate exception on file opening? > > So maybe you would like some other syntax? For example > > with from obj: > BLOCK > > That form can use magic methods with names different than > __enter__/__exit__ (and get rid of "with (yield from lock)" BTW). And > it's obviously? points on two suspending points: at BLOCK enter and > exit. > > On Mon, Apr 7, 2014 at 2:12 AM, Guido van Rossum wrote: > > I prefer to use a try/finally statement. One of the points of using > > yield-from is that you can always tell where your code may be suspended > by > > searching for "yield from". With your proposed change that would no > longer > > be true -- any with statement would also have to be inspected, and there > > would no longer be a way to know from the source alone whether it might > > yield or not (because it would dynamic -- there's no way to be sure at > > compile time which context manager is being used). > > > > > > On Sun, Apr 6, 2014 at 12:02 PM, Andrew Svetlov < > andrew.svetlov at gmail.com> > > wrote: > >> > >> Literally it may be generator itself or function that returns generator > >> object. > >> > >> Now I'm working on postgres library for asyncio > >> (http://aiopg.readthedocs.org/). > >> > >> And I would to use *with statement* for transaction handling like: > >> > >> with (yield from cursor.transaction()): > >> yield from cursor.execute(sql) > >> > >> The problem is: at exit of *with statement* I need to call `yield from > >> cursor.execute('COMMIT')` or `yield from cursor.execute('ROLLBACK')`. > >> > >> I can do it only in __exit__ in *context manager*, but python > >> understand only if __exit__: > >> - returns true value, that suppresses exception from code block > >> - returns None or any false value to propagate exception if any > >> - raises exception itself > >> > >> I propose to add new rule: > >> IF the code object is generator (co_flags & CO_GENERATOR) and __exit__ > >> returns generator object (isinstance(ret, types.GeneratorType)) > >> THEN do `yield from ret`. > >> > >> That's work fine if __exit__ itself is a *generator function* > >> (contains `yield` or `yield from` statements): call to *generator > >> function* returns *generator object*. > >> > >> The proposal: > >> 1. Doesn't break any existing code except if user accidentally > >> returns generator object instead of True from __exit__ call (he should > >> not to do this and I sure this is very rare case). > >> 2. Don't requires new syntax. > >> > >> asyncio itself uses: > >> > >> with (yield from lock): > >> BLOCK > >> > >> for locking etc but unlocking for asyncio objects doesn't requires any > >> `yield from`, so __exit__ code is just plain function but not > >> generator. > >> > >> Also I can live with asyncio trick for __enter__: > >> https://code.google.com/p/tulip/source/browse/asyncio/locks.py#156 > >> The way is a but annoying but unrolling a value returned by __enter__ > >> if the value is generator object will break existing code, sure. > >> > >> Thoughts? > >> > >> > >> -- > >> Thanks, > >> Andrew Svetlov > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > -- > Thanks, > Andrew Svetlov > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Mon Apr 7 03:12:09 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 6 Apr 2014 18:12:09 -0700 Subject: [Python-ideas] Relax __exit__ method to be a generator In-Reply-To: References: Message-ID: On Sun, Apr 6, 2014 at 6:04 PM, Guido van Rossum wrote: > Well, for new syntax, the bar is pretty high... And we should probably move > to python-ideas. And Guido's time machine strikes again! ;) -- Devin > On Sun, Apr 6, 2014 at 4:44 PM, Andrew Svetlov > wrote: >> >> Well, this is good point.. >> >> But from my experience try/finally is a common point of errors. >> Beginners use "with" properly as: >> >> with open(filename) as f: >> f.read() >> >> but switch to try/finally produces very common error. >> >> Instead of >> >> f = open(filename) >> try: >> f.read() >> finally: >> f.close() >> >> people usually write >> >> try: >> f = open(filename) >> f.read() >> finally: >> f.close() >> >> I saw it constantly many times. When I wrote an article about the >> problem and true solution in my blog I got several emails from my >> readers: please, tell me again why I need to move open() out of try >> block if it can generate exception on file opening? >> >> So maybe you would like some other syntax? For example >> >> with from obj: >> BLOCK >> >> That form can use magic methods with names different than >> __enter__/__exit__ (and get rid of "with (yield from lock)" BTW). And >> it's obviously? points on two suspending points: at BLOCK enter and >> exit. >> >> On Mon, Apr 7, 2014 at 2:12 AM, Guido van Rossum wrote: >> > I prefer to use a try/finally statement. One of the points of using >> > yield-from is that you can always tell where your code may be suspended >> > by >> > searching for "yield from". With your proposed change that would no >> > longer >> > be true -- any with statement would also have to be inspected, and there >> > would no longer be a way to know from the source alone whether it might >> > yield or not (because it would dynamic -- there's no way to be sure at >> > compile time which context manager is being used). >> > >> > >> > On Sun, Apr 6, 2014 at 12:02 PM, Andrew Svetlov >> > >> > wrote: >> >> >> >> Literally it may be generator itself or function that returns generator >> >> object. >> >> >> >> Now I'm working on postgres library for asyncio >> >> (http://aiopg.readthedocs.org/). >> >> >> >> And I would to use *with statement* for transaction handling like: >> >> >> >> with (yield from cursor.transaction()): >> >> yield from cursor.execute(sql) >> >> >> >> The problem is: at exit of *with statement* I need to call `yield from >> >> cursor.execute('COMMIT')` or `yield from cursor.execute('ROLLBACK')`. >> >> >> >> I can do it only in __exit__ in *context manager*, but python >> >> understand only if __exit__: >> >> - returns true value, that suppresses exception from code block >> >> - returns None or any false value to propagate exception if any >> >> - raises exception itself >> >> >> >> I propose to add new rule: >> >> IF the code object is generator (co_flags & CO_GENERATOR) and __exit__ >> >> returns generator object (isinstance(ret, types.GeneratorType)) >> >> THEN do `yield from ret`. >> >> >> >> That's work fine if __exit__ itself is a *generator function* >> >> (contains `yield` or `yield from` statements): call to *generator >> >> function* returns *generator object*. >> >> >> >> The proposal: >> >> 1. Doesn't break any existing code except if user accidentally >> >> returns generator object instead of True from __exit__ call (he should >> >> not to do this and I sure this is very rare case). >> >> 2. Don't requires new syntax. >> >> >> >> asyncio itself uses: >> >> >> >> with (yield from lock): >> >> BLOCK >> >> >> >> for locking etc but unlocking for asyncio objects doesn't requires any >> >> `yield from`, so __exit__ code is just plain function but not >> >> generator. >> >> >> >> Also I can live with asyncio trick for __enter__: >> >> https://code.google.com/p/tulip/source/browse/asyncio/locks.py#156 >> >> The way is a but annoying but unrolling a value returned by __enter__ >> >> if the value is generator object will break existing code, sure. >> >> >> >> Thoughts? >> >> >> >> >> >> -- >> >> Thanks, >> >> Andrew Svetlov >> >> _______________________________________________ >> >> Python-ideas mailing list >> >> Python-ideas at python.org >> >> https://mail.python.org/mailman/listinfo/python-ideas >> >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > >> > >> > >> > >> > -- >> > --Guido van Rossum (python.org/~guido) >> >> >> >> -- >> Thanks, >> Andrew Svetlov > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From njs at pobox.com Mon Apr 7 03:51:52 2014 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 7 Apr 2014 02:51:52 +0100 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: <20140406232038.GG16466@ando> References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: On Mon, Apr 7, 2014 at 12:20 AM, Steven D'Aprano wrote: > On Sun, Apr 06, 2014 at 11:02:00PM +0100, Nathaniel Smith wrote: >> On Sun, Mar 16, 2014 at 1:05 PM, Nick Coghlan wrote: >> > Aye, but at the moment it makes sense to wait and see if there is even >> > an argument to be had - Nathaniel may decide that even after reviewing >> > the question seriously, he doesn't want to propose a right associative >> > operator :) >> >> And indeed, that does seem to be the way things have worked out :-). >> http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069834.html > > That's a shame in one way -- for anyone using operator overloading to > define their own DSL, there's only ** if you want a right-associative > operator to overload. Still, that's an extremely marginal, and > hypothetical, use-case. Left-associative it is. > > Was that the last blocker for the PEP? Pretty much. Just posted to python-dev: https://mail.python.org/pipermail/python-dev/2014-April/133791.html -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From ncoghlan at gmail.com Tue Apr 8 05:18:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 8 Apr 2014 13:18:49 +1000 Subject: [Python-ideas] Relax __exit__ method to be a generator In-Reply-To: References: Message-ID: On 6 Apr 2014 21:13, "Devin Jeanpierre" wrote: > > On Sun, Apr 6, 2014 at 6:04 PM, Guido van Rossum wrote: > > Well, for new syntax, the bar is pretty high... And we should probably move > > to python-ideas. > > And Guido's time machine strikes again! ;) Also relevant to this topic: http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programming.html#additional-asynchronous-syntax There's a review of what does and doesn't currently work before that last subsection. (Short version: I'm currently +0 on syntax that allows asynchronous transactions and comprehensions, but would need to see some before/after comparisons with real asyncio code to get me to a +1) Cheers, Nick. P.S. I put the time machine keys back where I found them ;) > > -- Devin > > > On Sun, Apr 6, 2014 at 4:44 PM, Andrew Svetlov > > wrote: > >> > >> Well, this is good point.. > >> > >> But from my experience try/finally is a common point of errors. > >> Beginners use "with" properly as: > >> > >> with open(filename) as f: > >> f.read() > >> > >> but switch to try/finally produces very common error. > >> > >> Instead of > >> > >> f = open(filename) > >> try: > >> f.read() > >> finally: > >> f.close() > >> > >> people usually write > >> > >> try: > >> f = open(filename) > >> f.read() > >> finally: > >> f.close() > >> > >> I saw it constantly many times. When I wrote an article about the > >> problem and true solution in my blog I got several emails from my > >> readers: please, tell me again why I need to move open() out of try > >> block if it can generate exception on file opening? > >> > >> So maybe you would like some other syntax? For example > >> > >> with from obj: > >> BLOCK > >> > >> That form can use magic methods with names different than > >> __enter__/__exit__ (and get rid of "with (yield from lock)" BTW). And > >> it's obviously? points on two suspending points: at BLOCK enter and > >> exit. > >> > >> On Mon, Apr 7, 2014 at 2:12 AM, Guido van Rossum wrote: > >> > I prefer to use a try/finally statement. One of the points of using > >> > yield-from is that you can always tell where your code may be suspended > >> > by > >> > searching for "yield from". With your proposed change that would no > >> > longer > >> > be true -- any with statement would also have to be inspected, and there > >> > would no longer be a way to know from the source alone whether it might > >> > yield or not (because it would dynamic -- there's no way to be sure at > >> > compile time which context manager is being used). > >> > > >> > > >> > On Sun, Apr 6, 2014 at 12:02 PM, Andrew Svetlov > >> > > >> > wrote: > >> >> > >> >> Literally it may be generator itself or function that returns generator > >> >> object. > >> >> > >> >> Now I'm working on postgres library for asyncio > >> >> (http://aiopg.readthedocs.org/). > >> >> > >> >> And I would to use *with statement* for transaction handling like: > >> >> > >> >> with (yield from cursor.transaction()): > >> >> yield from cursor.execute(sql) > >> >> > >> >> The problem is: at exit of *with statement* I need to call `yield from > >> >> cursor.execute('COMMIT')` or `yield from cursor.execute('ROLLBACK')`. > >> >> > >> >> I can do it only in __exit__ in *context manager*, but python > >> >> understand only if __exit__: > >> >> - returns true value, that suppresses exception from code block > >> >> - returns None or any false value to propagate exception if any > >> >> - raises exception itself > >> >> > >> >> I propose to add new rule: > >> >> IF the code object is generator (co_flags & CO_GENERATOR) and __exit__ > >> >> returns generator object (isinstance(ret, types.GeneratorType)) > >> >> THEN do `yield from ret`. > >> >> > >> >> That's work fine if __exit__ itself is a *generator function* > >> >> (contains `yield` or `yield from` statements): call to *generator > >> >> function* returns *generator object*. > >> >> > >> >> The proposal: > >> >> 1. Doesn't break any existing code except if user accidentally > >> >> returns generator object instead of True from __exit__ call (he should > >> >> not to do this and I sure this is very rare case). > >> >> 2. Don't requires new syntax. > >> >> > >> >> asyncio itself uses: > >> >> > >> >> with (yield from lock): > >> >> BLOCK > >> >> > >> >> for locking etc but unlocking for asyncio objects doesn't requires any > >> >> `yield from`, so __exit__ code is just plain function but not > >> >> generator. > >> >> > >> >> Also I can live with asyncio trick for __enter__: > >> >> https://code.google.com/p/tulip/source/browse/asyncio/locks.py#156 > >> >> The way is a but annoying but unrolling a value returned by __enter__ > >> >> if the value is generator object will break existing code, sure. > >> >> > >> >> Thoughts? > >> >> > >> >> > >> >> -- > >> >> Thanks, > >> >> Andrew Svetlov > >> >> _______________________________________________ > >> >> Python-ideas mailing list > >> >> Python-ideas at python.org > >> >> https://mail.python.org/mailman/listinfo/python-ideas > >> >> Code of Conduct: http://python.org/psf/codeofconduct/ > >> > > >> > > >> > > >> > > >> > -- > >> > --Guido van Rossum (python.org/~guido) > >> > >> > >> > >> -- > >> Thanks, > >> Andrew Svetlov > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From epsilonmichael at gmail.com Tue Apr 8 23:09:57 2014 From: epsilonmichael at gmail.com (Michael Mitchell) Date: Tue, 8 Apr 2014 14:09:57 -0700 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: I haven't been following this thread too closely, so please stop me if this has been covered, but has overloading the @ operator as function composition been considered yet? An example would be filter(a @ b, lst) as opposed to filter(lambda x: a(b(x)), lst) On Sun, Apr 6, 2014 at 6:51 PM, Nathaniel Smith wrote: > On Mon, Apr 7, 2014 at 12:20 AM, Steven D'Aprano > wrote: > > On Sun, Apr 06, 2014 at 11:02:00PM +0100, Nathaniel Smith wrote: > >> On Sun, Mar 16, 2014 at 1:05 PM, Nick Coghlan > wrote: > >> > Aye, but at the moment it makes sense to wait and see if there is even > >> > an argument to be had - Nathaniel may decide that even after reviewing > >> > the question seriously, he doesn't want to propose a right associative > >> > operator :) > >> > >> And indeed, that does seem to be the way things have worked out :-). > >> > http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069834.html > > > > That's a shame in one way -- for anyone using operator overloading to > > define their own DSL, there's only ** if you want a right-associative > > operator to overload. Still, that's an extremely marginal, and > > hypothetical, use-case. Left-associative it is. > > > > Was that the last blocker for the PEP? > > Pretty much. Just posted to python-dev: > https://mail.python.org/pipermail/python-dev/2014-April/133791.html > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Apr 8 23:17:08 2014 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 8 Apr 2014 22:17:08 +0100 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: It's been raised a few times, but the problem is that there's no evidence that anyone actually needs a short way to perform composition - notice the composition operation's never even been added to functools. If you do want to make that case though then there's nothing stopping you :-) On 8 Apr 2014 22:10, "Michael Mitchell" wrote: > I haven't been following this thread too closely, so please stop me if > this has been covered, but has overloading the @ operator as function > composition been considered yet? > > An example would be > > filter(a @ b, lst) > > as opposed to > > filter(lambda x: a(b(x)), lst) > > > On Sun, Apr 6, 2014 at 6:51 PM, Nathaniel Smith wrote: > >> On Mon, Apr 7, 2014 at 12:20 AM, Steven D'Aprano >> wrote: >> > On Sun, Apr 06, 2014 at 11:02:00PM +0100, Nathaniel Smith wrote: >> >> On Sun, Mar 16, 2014 at 1:05 PM, Nick Coghlan >> wrote: >> >> > Aye, but at the moment it makes sense to wait and see if there is >> even >> >> > an argument to be had - Nathaniel may decide that even after >> reviewing >> >> > the question seriously, he doesn't want to propose a right >> associative >> >> > operator :) >> >> >> >> And indeed, that does seem to be the way things have worked out :-). >> >> >> http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069834.html >> > >> > That's a shame in one way -- for anyone using operator overloading to >> > define their own DSL, there's only ** if you want a right-associative >> > operator to overload. Still, that's an extremely marginal, and >> > hypothetical, use-case. Left-associative it is. >> > >> > Was that the last blocker for the PEP? >> >> Pretty much. Just posted to python-dev: >> https://mail.python.org/pipermail/python-dev/2014-April/133791.html >> >> -n >> >> -- >> Nathaniel J. Smith >> Postdoctoral researcher - Informatics - University of Edinburgh >> http://vorpus.org >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From epsilonmichael at gmail.com Wed Apr 9 01:00:45 2014 From: epsilonmichael at gmail.com (Michael Mitchell) Date: Tue, 8 Apr 2014 16:00:45 -0700 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: I'm not confident I can make that case very well, but I suppose I can try :). There do seem to be some hand-rolled composition functions in various Github repositories: https://github.com/search?l=python&q=%22def+compose%22&ref=searchresults&type=Code I'm not sure why a composition operator has never been added to the functools library, but I'd make the observation that a composition function wouldn't be much less verbose than in my earlier lambda example. A infix binary operator, however, would be quite more concise, especially so when chaining multiple functions. There's also precedent to the @ symbol being associated with functions, i.e. decorator syntax. Decorators are callables that accept callables and return callables, which parallels the infix definition of accepting two callables and returning a callable. On Tue, Apr 8, 2014 at 2:17 PM, Nathaniel Smith wrote: > It's been raised a few times, but the problem is that there's no evidence > that anyone actually needs a short way to perform composition - notice the > composition operation's never even been added to functools. If you do want > to make that case though then there's nothing stopping you :-) > On 8 Apr 2014 22:10, "Michael Mitchell" wrote: > >> I haven't been following this thread too closely, so please stop me if >> this has been covered, but has overloading the @ operator as function >> composition been considered yet? >> >> An example would be >> >> filter(a @ b, lst) >> >> as opposed to >> >> filter(lambda x: a(b(x)), lst) >> >> >> On Sun, Apr 6, 2014 at 6:51 PM, Nathaniel Smith wrote: >> >>> On Mon, Apr 7, 2014 at 12:20 AM, Steven D'Aprano >>> wrote: >>> > On Sun, Apr 06, 2014 at 11:02:00PM +0100, Nathaniel Smith wrote: >>> >> On Sun, Mar 16, 2014 at 1:05 PM, Nick Coghlan >>> wrote: >>> >> > Aye, but at the moment it makes sense to wait and see if there is >>> even >>> >> > an argument to be had - Nathaniel may decide that even after >>> reviewing >>> >> > the question seriously, he doesn't want to propose a right >>> associative >>> >> > operator :) >>> >> >>> >> And indeed, that does seem to be the way things have worked out :-). >>> >> >>> http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069834.html >>> > >>> > That's a shame in one way -- for anyone using operator overloading to >>> > define their own DSL, there's only ** if you want a right-associative >>> > operator to overload. Still, that's an extremely marginal, and >>> > hypothetical, use-case. Left-associative it is. >>> > >>> > Was that the last blocker for the PEP? >>> >>> Pretty much. Just posted to python-dev: >>> https://mail.python.org/pipermail/python-dev/2014-April/133791.html >>> >>> -n >>> >>> -- >>> Nathaniel J. Smith >>> Postdoctoral researcher - Informatics - University of Edinburgh >>> http://vorpus.org >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Apr 9 04:40:42 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 8 Apr 2014 19:40:42 -0700 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: On Apr 8, 2014, at 16:00, Michael Mitchell wrote: > I'm not confident I can make that case very well, but I suppose I can try :). > > There do seem to be some hand-rolled composition functions in various Github repositories: https://github.com/search?l=python&q=%22def+compose%22&ref=searchresults&type=Code > > I'm not sure why a composition operator has never been added to the functools library, but I'd make the observation that a composition function wouldn't be much less verbose than in my earlier lambda example. A infix binary operator, however, would be quite more concise, especially so when chaining multiple functions. A few problems have been raised with this. The big one is that it's not clear what it means to compose functions unless they take and return a single value. While you could make it call g(*f(x)) to pass multiple values, that makes it no longer work with single-value functions. Functional languages deal with that by having equally compact ways to do all kinds of other general higher-order tasks, like partialing functions and sectioning operators (or they just use currying to make all functions partial), flipping arguments, raising functions (like a map that returns a function over an iterable, instead of taking an iterable in the map call), etc. Python has nothing but partial, it's not that compact, and it's buried in functools. Also, some people find that code using higher-order functions more than necessary usually doesn't look very pythonic. All kinds of things that are done by building up a higher-order function and then calling it in a language like Haskell are usually instead done by chaining or imperatively calling functions in Python. This could lead to a "more than one way to do it" situation, or just to code that's harder to understand without thinking it through. (I personally don't find this argument very compelling, but then it's hard for me to put myself in the shoes of people who never use functional languages, so that may not mean much. And I think Guido is one of the people who finds it compelling.) Both left-compose and right-compose are perfectly valid operations, and the one that looks obviously right to novices is the opposite of the one that's obviously right (or at least more often) to experts. Finally, compose and matrix multiplication are false cognates. Matrices are operations on vectors under multiplication--the same operator used for composing those operations. Functions are operations on values under calling--a completely different operator than @. Finally, it would help to have a real use case instead of just filter(a @ b, x). Presumably b is some function from elements of x to some other type, and a is a predicate function from that type to bool... but it's hard to think of useful functions (as opposed to expressions you'd have to wrap in lambda anyway) that fit that bill off the top of my head. None of these means the idea is impossible or useless, just that it's less obvious a win than it looks at first. > There's also precedent to the @ symbol being associated with functions, i.e. decorator syntax. Decorators are callables that accept callables and return callables, which parallels the infix definition of accepting two callables and returning a callable. > > > On Tue, Apr 8, 2014 at 2:17 PM, Nathaniel Smith wrote: >> It's been raised a few times, but the problem is that there's no evidence that anyone actually needs a short way to perform composition - notice the composition operation's never even been added to functools. If you do want to make that case though then there's nothing stopping you :-) >> >> On 8 Apr 2014 22:10, "Michael Mitchell" wrote: >>> I haven't been following this thread too closely, so please stop me if this has been covered, but has overloading the @ operator as function composition been considered yet? >>> >>> An example would be >>> >>> filter(a @ b, lst) >>> >>> as opposed to >>> >>> filter(lambda x: a(b(x)), lst) >>> >>> >>> On Sun, Apr 6, 2014 at 6:51 PM, Nathaniel Smith wrote: >>>> On Mon, Apr 7, 2014 at 12:20 AM, Steven D'Aprano wrote: >>>> > On Sun, Apr 06, 2014 at 11:02:00PM +0100, Nathaniel Smith wrote: >>>> >> On Sun, Mar 16, 2014 at 1:05 PM, Nick Coghlan wrote: >>>> >> > Aye, but at the moment it makes sense to wait and see if there is even >>>> >> > an argument to be had - Nathaniel may decide that even after reviewing >>>> >> > the question seriously, he doesn't want to propose a right associative >>>> >> > operator :) >>>> >> >>>> >> And indeed, that does seem to be the way things have worked out :-). >>>> >> http://mail.scipy.org/pipermail/numpy-discussion/2014-April/069834.html >>>> > >>>> > That's a shame in one way -- for anyone using operator overloading to >>>> > define their own DSL, there's only ** if you want a right-associative >>>> > operator to overload. Still, that's an extremely marginal, and >>>> > hypothetical, use-case. Left-associative it is. >>>> > >>>> > Was that the last blocker for the PEP? >>>> >>>> Pretty much. Just posted to python-dev: >>>> https://mail.python.org/pipermail/python-dev/2014-April/133791.html >>>> >>>> -n >>>> >>>> -- >>>> Nathaniel J. Smith >>>> Postdoctoral researcher - Informatics - University of Edinburgh >>>> http://vorpus.org >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Apr 9 05:30:36 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Apr 2014 13:30:36 +1000 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: On Wed, Apr 9, 2014 at 9:00 AM, Michael Mitchell wrote: > I'm not sure why a composition operator has never been added to the > functools library, but I'd make the observation that a composition function > wouldn't be much less verbose than in my earlier lambda example. A infix > binary operator, however, would be quite more concise, especially so when > chaining multiple functions. Is "a @ b" equivalent to "lambda x: a(b(x))" or to "lambda x: b(a(x))"? Is there a sufficiently-obvious way to make it clear that the one on the right gets called first? ChrisA From guido at python.org Wed Apr 9 05:33:30 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 8 Apr 2014 23:33:30 -0400 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: The question is academic. Regardless of which answer you choose that is not going to happen. On Tue, Apr 8, 2014 at 11:30 PM, Chris Angelico wrote: > On Wed, Apr 9, 2014 at 9:00 AM, Michael Mitchell > wrote: > > I'm not sure why a composition operator has never been added to the > > functools library, but I'd make the observation that a composition > function > > wouldn't be much less verbose than in my earlier lambda example. A infix > > binary operator, however, would be quite more concise, especially so when > > chaining multiple functions. > > Is "a @ b" equivalent to "lambda x: a(b(x))" or to "lambda x: > b(a(x))"? Is there a sufficiently-obvious way to make it clear that > the one on the right gets called first? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Apr 9 05:52:16 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Apr 2014 13:52:16 +1000 Subject: [Python-ideas] [RFC] draft PEP: Dedicated infix operators for matrix multiplication and matrix power In-Reply-To: References: <53238DB1.7060608@canterbury.ac.nz> <20140315073508.GB16526@ando> <20140315122733.35ed7909@fsol> <20140315151143.GE16526@ando> <20140316132634.13d6d589@fsol> <20140406232038.GG16466@ando> Message-ID: On Wed, Apr 9, 2014 at 1:33 PM, Guido van Rossum wrote: > The question is academic. Regardless of which answer you choose that is not > going to happen. > > On Tue, Apr 8, 2014 at 11:30 PM, Chris Angelico wrote: >> >> Is "a @ b" equivalent to "lambda x: a(b(x))" or to "lambda x: >> b(a(x))"? Is there a sufficiently-obvious way to make it clear that >> the one on the right gets called first? Well there you go, that settles it :) ChrisA From barry at python.org Thu Apr 10 17:05:30 2014 From: barry at python.org (Barry Warsaw) Date: Thu, 10 Apr 2014 11:05:30 -0400 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. References: Message-ID: <20140410110530.4c8dd290@anarchist.localdomain> Nice PEP; I'm sorry we didn't get a chance to discuss it at the language summit. On Apr 05, 2014, at 11:43 PM, Eric Snow wrote: >Alternate Approaches >==================== > >Opt-out Decorator >----------------- > >This is identical to the current proposal with the exception that Python >would also provide a decorator in functools that would cause collected >keyword arguments to be packed into a normal dict instead of an >OrderedDict. > >Prognosis: > >This would only be necessary if performance is determined to be >significantly different in some uncommon cases or that there are other >backward-compatibility concerns that cannot be resolved otherwise. > >Opt-in Decorator >---------------- > >The status quo would be unchanged. Instead Python would provide a >decorator in functools that would register or mark the decorated >function as one that should get ordered keyword arguments. The >performance overhead to check the function at call time would be >marginal. > >Prognosis: > >The only real down-side is in the case of function wrappers factories >(e.g. functools.partial and many decorators) that aim to perfectly >preserve keyword arguments by using kwargs in the wrapper definition >and kwargs unpacking in the call to the wrapped function. Each wrapper >would have to be updated separately, though having functools.wraps() do >this automaticallywould help. Since this is touching a fundamental and common aspect of the language, I think a lot of analysis has to be done for both performance and semantics if the feature is opt-out. People do crazy things and even small performance hits add up. The use cases are valid but rare I submit, so I think most Python code doesn't care. Thus I'd lean toward an opt-in approach. Perhaps a transition period makes sense. Add both opt-in and opt-out decorators, implement opt-in by default for Python 3.5 with a -X option to switch it to opt-out. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From ericsnowcurrently at gmail.com Thu Apr 10 18:00:06 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 10 Apr 2014 10:00:06 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: <20140410110530.4c8dd290@anarchist.localdomain> References: <20140410110530.4c8dd290@anarchist.localdomain> Message-ID: On Thu, Apr 10, 2014 at 9:05 AM, Barry Warsaw wrote: > Nice PEP; I'm sorry we didn't get a chance to discuss it at the language > summit. Yeah, I didn't even bother mentioning it because I didn't have any performance numbers yet. I Hope to have some today, however. > Since this is touching a fundamental and common aspect of the language, I > think a lot of analysis has to be done for both performance and semantics if > the feature is opt-out. The proposal currently doesn't even involve opting anything. :) Opt-out is my fallback proposal in case performance has significant change when "people do crazy things". > People do crazy things and even small performance > hits add up. The use cases are valid but rare I submit, so I think most > Python code doesn't care. Thus I'd lean toward an opt-in approach. The problem is that opt-in causes all sorts of complication due to some wrapper factories like functools.partial and many decorators. They take **kwargs, and then unpack them in the wrapped call, so the wrapper definition would have to know how to opt-in when the wrappee has opted in. Andrew Barnert and I discussed this at length in the previous python-ideas thread. My conclusion was that we should avoid opt-in if at all possible. > > Perhaps a transition period makes sense. Add both opt-in and opt-out > decorators, implement opt-in by default for Python 3.5 with a -X option to > switch it to opt-out. If we could work out the opt-in complications that could work. However, I just don't think opt-in will be worth the trouble. That said, right now we're mostly just guessing about the performance impact. :) I hope to have a better answer to that question soon. -eric From dimaqq at gmail.com Thu Apr 10 19:57:22 2014 From: dimaqq at gmail.com (Dima Tisnek) Date: Thu, 10 Apr 2014 19:57:22 +0200 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: Sorry to be a spoil-sport, but what next? Preserve order of globals? Preserve order of locals? Preserve order of class members? methinks the whole point of keyword arguments is that they are named. order is a much weaker concept. d; my 2c. On 6 April 2014 07:43, Eric Snow wrote: > Here's the proper proposal I promised. Hopefully it reflects our > discussions and the past ones too. My goal here is to focus in on > what I consider to be the most viable approach, which capturing the > gist of the various concerns and alternatives. There are still a few > small gaps that I will work on as time permits. Feedback is welcome. > > FYI, I also hope to have a basic implementation up in time for the > language summit at PyCon (Wednesday) so that related discussions there > might have something more concrete to support them. :) > > -eric > > ===================================================================== > > PEP: XXX > Title: Preserving the order of \*\*kwargs in a function. > Version: $Revision$ > Last-Modified: $Date$ > Author: Eric Snow > Discussions-To: python-ideas at python.org > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 5-Apr-2014 > Python-Version: 3.5 > Post-History: > Resolution: > > > Abstract > ======== > > The \*\*kwargs syntax in a function definition indicates that the > interpreter should collect all keyword arguments that do not correspond > to other named parameters. However, > Python does not preserved the order in which those collected keyword > arguments were passed to the function. In some contexts the order > matters. This PEP introduces a mechanism by which the passed order of > collected keyword arguments will now be preserved. > > > Motivation > ========== > > Python's \*\*kwargs syntax in function definitions provides a powerful > means of dynamically handling keyword arguments. In some applications > of the syntax (see _`Use Cases`), the semantics applied to the > collected keyword arguments requires that order be preserved. > Unsurprisingly, this is similar to how OrderedDict is related to dict. > > Currently to preserved the order you have to do so manually and > separately from the actual function call. This involves building an > ordered mapping, whether an OrderedDict or an iterable of 2-tuples, > which is then passed as a single argument to the function. > [#arg_unpacking]_ > > With the capability described in this PEP, that boilerplate would no > longer be required. > > For comparision, currently:: > > kwargs = OrderedDict() > kwargs['eggs'] = ... > ... > def spam(a, kwargs): > ... > > and with this proposal:: > > def spam(a, **kwargs): > ... > > Nick Coglan, speaking of some of the uses cases, summed it up well > [#nick_obvious]_:: > > These *can* all be done today, but *not* by using keyword arguments. > In my view, the problem to be addressed is that keyword arguments > *look* like they should work for these cases, because they have a > definite order in the source code. The only reason they don't work > is because the interpreter throws that ordering information away. > > It's a textbook case of a language feature becoming an attractive > nuisance in some circumstances: the simple and obvious solution for > the above use cases *doesn't actually work* for reasons that aren't > obviously clear if you don't have a firm grasp of Python's admittedly > complicated argument handling. > > This observation is supported by the appearance of this proposal over > the years and the numerous times that people have been confused by the > constructor for OrderedDict. [#past_threads]_ [#loss_of_order]_ > [#compact_dict]_ > > > Use Cases > ========= > > As Nick noted, the current behavior of \*\*kwargs is unintuitive in > cases where one would expect order to matter. Aside from more specific > cases outlined below, in general "anything else where you want to > control the iteration order *and* set field names and values in a single > call will potentially benefit." [#nick_general]_ That matters in the > case of factories (e.g. __init__()) for ordered types. > > Serialization > ------------- > > Obviously OrderedDict would benefit (both __init__() and update()) from > ordered kwargs. However, the benefit also extends to serialization > APIs [#nick_obvious]_:: > > In the context of serialisation, one key lesson we have learned is > that arbitrary ordering is a problem when you want to minimise > spurious diffs, and sorting isn't a simple solution. > > Tools like doctest don't tolerate spurious diffs at all, but are > often amenable to a sorting based answer. > > The cases where it would be highly desirable to be able use keyword > arguments to control the order of display of a collection of key > value pairs are ones like: > > * printing out key:value pairs in CLI output > * mapping semantic names to column order in a CSV > * serialising attributes and elements in particular orders in XML > * serialising map keys in particular orders in human readable formats > like JSON and YAML (particularly when they're going to be placed > under source control) > > Debugging > --------- > > In the words of Raymond Hettinger [#raymond_debug]_:: > > It makes it easier to debug if the arguments show-up in the order > they were created. AFAICT, no purpose is served by scrambling them. > > Other Use Cases > --------------- > > * Mock objects. [#mock]_ > * Controlling object presentation. > * Alternate namedtuple() where defaults can be specified. > * Specifying argument priority by order. > > > Concerns > ======== > > Performance > ----------- > > As already noted, the idea of ordered keyword arguments has come up on > a number of occasions. Each time it has been met with the same > response, namely that preserving keyword arg order would have a > sufficiently adverse effect on function call performance that it's not > worth doing. However, Guido noted the following [#guido_open]_:: > > Making **kwds ordered is still open, but requires careful design and > implementation to avoid slowing down function calls that don't benefit. > > As will be noted below, there are ways to work around this at the > expense of increased complication. Ultimately the simplest approach is > the one that makes the most sense: pack collected key word arguments > into an OrderedDict. However, without a C implementation of OrderedDict > there isn't much to discuss. That should change in Python 3.5. > [#c_ordereddict]_ > > In some cases the difference of performance between dict and OrderedDict > *may* be of significance. For instance: when the collected > kwargs has an extended lifetime outside the originating function or the > number of collected kwargs is massive. However, the difference in > performance (both CPU and memory) in those cases should not be > significant. Furthermore, the performance of the C OrderedDict > implementation is essentially identical with dict for the non-mutating > API. A concrete representation of the difference in performance will be > a part of this proposal before its resolution. > > Other Python Implementations > ---------------------------- > > Another important issue to consider is that new features must be > cognizant of the multiple Python implementations. At some point each of > them would be expected to have implemented ordered kwargs. In this > regard there doesn't seem to be an issue with the idea. [#ironpython]_ > Each of the major Python implementations will be consulted regarding > this proposal before its resolution. > > > Specification > ============= > > Starting in version 3.5 Python will preserve the order of keyword > arguments as passed to a function. This will apply only to functions > for which the definition uses the \*\*kwargs syntax for collecting > otherwise unspecified keyword arguments. Only the order of those > keyword arguments will be preserved. > > Relationship to **-unpacking syntax > ----------------------------------- > > The ** unpacking syntax in function calls has no special connection with > this proposal. Keyword arguments provided by unpacking will be treated > in exactly the same way as they are now: ones that match defined > parameters are gather there and the remainder will be collected into the > ordered kwargs (just like any other unmatched keyword argument). > > Note that unpacking a mapping with undefined order, such as dict, will > preserve its iteration order like normal. It's just that the order will > remain undefined. The OrderedDict into which the unpacked key-value > pairs will then be packed will not be able to provide any alternate > ordering. This should not be surprising. > > There have been brief discussions of simply passing these mappings > through to the functions kwargs without unpacking and repacking them, > but that is both outside the scope of this proposal and probably a bad > idea regardless. (There is a reason those discussions were brief.) > > Relationship to inspect.Signature > --------------------------------- > > Signature objects should need no changes. The `kwargs` parameter of > inspect.BoundArguments (returned by Signature.bind() and > Signature.bind_partial()) will change from a dict to an OrderedDict. > > C-API > ----- > > TBD > > Syntax > ------ > > No syntax is added or changed by this proposal. > > Backward-Compatibility > ---------------------- > > The following will change: > > * type(kwargs) > * iteration order of kwargs will now be consistent (except of course in > the case described above) > * as already noted, performance will be marginally different > > None of these should be an issue. However, each will be carefully > considered while this proposal is under discussion. > > > Reference Implementation > ======================== > > TBD > > Implementation Notes > -------------------- > > TBD > > > Alternate Approaches > ==================== > > Opt-out Decorator > ----------------- > > This is identical to the current proposal with the exception that Python > would also provide a decorator in functools that would cause collected > keyword arguments to be packed into a normal dict instead of an > OrderedDict. > > Prognosis: > > This would only be necessary if performance is determined to be > significantly different in some uncommon cases or that there are other > backward-compatibility concerns that cannot be resolved otherwise. > > Opt-in Decorator > ---------------- > > The status quo would be unchanged. Instead Python would provide a > decorator in functools that would register or mark the decorated > function as one that should get ordered keyword arguments. The > performance overhead to check the function at call time would be > marginal. > > Prognosis: > > The only real down-side is in the case of function wrappers factories > (e.g. functools.partial and many decorators) that aim to perfectly > preserve keyword arguments by using kwargs in the wrapper definition > and kwargs unpacking in the call to the wrapped function. Each wrapper > would have to be updated separately, though having functools.wraps() do > this automaticallywould help. > > __kworder__ > ----------- > > The order of keyword arguments would be stored separately in a list at > call time. The list would be bound to __kworder__ in the function > locals. > > Prognosis: > > This likewise complicates the wrapper case. > > Compact dict with faster iteration > ---------------------------------- > > Raymond Hettinger has introduced the idea of a dict implementation that > would result in preserving insertion order on dicts (until the first > deletion). This would be a perfect fit for kwargs. [#compact_dict]_ > > Prognosis: > > The idea is still uncertain in both viability and timeframe. > > ***kwargs > --------- > > This would add a new form to a function's signature as a mutually > exclusive parallel to \*\*kwargs. The new syntax, ***kwargs (note that > there are three asterisks), would indicate that kwargs should preserve > the order of keyword arguments. > > Prognosis: > > New syntax is only added to Python under the most *dire* circumstances. > With other available solutions, new syntax is not justifiable. > Furthermore, like all opt-in solutions, the new syntax would complicate > the pass-through case. > > annotations > ----------- > > This is a variation on the decorator approach. Instead of using a > decorator to mark the function, you would use a function annotation on > \*\*kwargs. > > Prognosis: > > In addition to the pass-through complication, annotations have been > actively discouraged in Python core development. Use of annotations to > opt-in to order preservation runs the risk of interfering with other > application-level use of annotations. > > dict.__order__ > -------------- > > dict objects would have a new attribute, `__order__` that would default > to None and that in the kwargs case the interpreter would use in the > same way as described above for __kworder__. > > Prognosis: > > It would mean zero impact on kwargs performance but the change would be > pretty intrusive (Python uses dict a lot). Also, for the wrapper case > the interpreter would have to be careful to preserve `__order__`. > > KWArgsDict.__order__ > -------------------- > > This is the same as the `dict.__order__` idea, but kwargs would be an > instance of a new minimal dict subclass that provides the `__order__` > attribute. dict would instead be unchanged. > > Prognosis: > > Simply switching to OrderedDict is a less complicated and more intuitive > change. > > > Acknowledgements > ================ > > Thanks to Andrew Barnert for helpful feedback and to the participants of > all the past email threads. > > > Footnotes > ========= > > .. [#arg_unpacking] > > Alternately, you could also replace ** in your function definition > with * and then pass in key/value 2-tuples. This has the advantage > of not requiring the keys to be valid identifier strings. See > https://mail.python.org/pipermail/python-ideas/2014-April/027491.html. > > > References > ========== > > .. [#nick_obvious] > https://mail.python.org/pipermail/python-ideas/2014-April/027512.html > > .. [#past_threads] > https://mail.python.org/pipermail/python-ideas/2009-April/004163.html > https://mail.python.org/pipermail/python-ideas/2010-October/008445.html > https://mail.python.org/pipermail/python-ideas/2011-January/009037.html > https://mail.python.org/pipermail/python-ideas/2013-February/019690.html > https://mail.python.org/pipermail/python-ideas/2013-May/020727.html > https://mail.python.org/pipermail/python-ideas/2014-March/027225.html > http://bugs.python.org/issue16276 > http://bugs.python.org/issue16553 > http://bugs.python.org/issue19026 > http://bugs.python.org/issue5397#msg82972 > > .. [#loss_of_order] > https://mail.python.org/pipermail/python-dev/2007-February/071310.html > > .. [#compact_dict] > https://mail.python.org/pipermail/python-dev/2012-December/123028.html > https://mail.python.org/pipermail/python-dev/2012-December/123105.html > https://mail.python.org/pipermail/python-dev/2013-May/126327.html > https://mail.python.org/pipermail/python-dev/2013-May/126328.html > > .. [#nick_general] > https://mail.python.org/pipermail/python-dev/2012-December/123105.html > > .. [#raymond_debug] > https://mail.python.org/pipermail/python-dev/2013-May/126327.html > > .. [#mock] > https://mail.python.org/pipermail/python-ideas/2009-April/004163.html > https://mail.python.org/pipermail/python-ideas/2009-April/004165.html > https://mail.python.org/pipermail/python-ideas/2009-April/004175.html > > .. [guido_open] > https://mail.python.org/pipermail/python-dev/2013-May/126404.html > > .. [#c_ordereddict] > http://bugs.python.org/issue16991 > > .. [#ironpython] > https://mail.python.org/pipermail/python-dev/2012-December/123100.html > > > Copyright > ========= > > This document has been placed in the public domain. > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From njs at pobox.com Thu Apr 10 20:09:41 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 10 Apr 2014 19:09:41 +0100 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: On Thu, Apr 10, 2014 at 6:57 PM, Dima Tisnek wrote: > Sorry to be a spoil-sport, but what next? > > Preserve order of globals? > Preserve order of locals? > Preserve order of class members? But these are all currently possible... PEP 3115 explicitly mentions preserving order of class members as a motivation for __prepare__. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From ethan at stoneleaf.us Thu Apr 10 20:08:34 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 10 Apr 2014 11:08:34 -0700 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: <5346DE22.20808@stoneleaf.us> On 04/10/2014 10:57 AM, Dima Tisnek wrote: > > Preserve order of class members? It is now possible to do this with a meta class. -- ~Ethan~ From ericsnowcurrently at gmail.com Thu Apr 10 20:38:34 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 10 Apr 2014 12:38:34 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: Message-ID: On Apr 10, 2014 1:57 PM, "Dima Tisnek" wrote: > > Sorry to be a spoil-sport, but what next? Oh this isn't so bad :) > > Preserve order of globals? Already do-able. > Preserve order of locals? Function execution stands alone in Python in how explicitly we restrict flexibility to customize (for the sake of performance). However, see the next point. > Preserve order of class members? As already noted, you can do so now. Furthermore, you can use that mechanism to preserve the order of function "locals". > > methinks the whole point of keyword arguments is that they are named. > > order is a much weaker concept. I agree it is weaker. I still think it is worth preserving, as explained in the PEP. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu Apr 10 22:18:07 2014 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 10 Apr 2014 16:18:07 -0400 Subject: [Python-ideas] [RFC] Draft-PEP: Adding structured data to built-in exceptions In-Reply-To: References: Message-ID: <5346FC7F.6030400@trueblade.com> On 04/06/2014 01:32 PM, Sebastian Kreft wrote: > Continuing the discussion started > in https://mail.python.org/pipermail/python-ideas/2014-February/025601.html, > attached is a draft PEP explaining the rationale of the idea and the > required changes. > > I'm looking forward to your comments. I'm generally in favor. My only concern would be object lifetimes if you're holding references to actual objects. For motivating use cases, one might be the difficulty in a string parser getting this correct: >>> {}["fo\"\\'\""] Traceback (most recent call last): File "", line 1, in KeyError: 'fo"\\\'"' From jacek.pliszka at gmail.com Thu Apr 10 23:12:14 2014 From: jacek.pliszka at gmail.com (Jacek Pliszka) Date: Thu, 10 Apr 2014 23:12:14 +0200 Subject: [Python-ideas] map, filter, reduce methods for generators Message-ID: Hi! Sorry if someone has already talked about it (my simple search did not show any results). What do you think about adding map, flatmap, filter and reduce methods to generator type ? I must admit I've seen and I like Java 8 notation and I think it might be more readable than Python way in a few occasions. I would like to be able to write: range(100).\ filter( f1 ).\ map( f2 ).\ filter( f3 ).\ map( f4 ).\ reduce(operator.add) in addition to current Pythonic way of reduce( operator.add, f4(x) for x in ( f2(y) for y in range(100) if f1(y)) if f3(x) ) Though longer - Java way seems to be a bit more readable as the notation follows the data flow sequence. Do you think it is worth a PEP? BR, Jacek -------------- next part -------------- An HTML attachment was scrubbed... URL: From haoyi.sg at gmail.com Thu Apr 10 23:39:05 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Thu, 10 Apr 2014 14:39:05 -0700 Subject: [Python-ideas] map, filter, reduce methods for generators In-Reply-To: References: Message-ID: I think it's a great idea that probably won't have much uptake on the list; FWIW in Scala you'd write // Sum of the squares of all odd numbers up to a hundred *(0 until 100).filter(_ % 2 == 1)* * .map(math.pow(_, 2))* * .reduce(_ + _)* But method chaining isn't really a thing in the python world, and people don't seem to like it as much as I do. On Thu, Apr 10, 2014 at 2:12 PM, Jacek Pliszka wrote: > Hi! > > Sorry if someone has already talked about it (my simple search did not > show any results). > > What do you think about adding map, flatmap, filter and reduce methods to > generator type ? > > I must admit I've seen and I like Java 8 notation and I think it might be > more readable than Python way in a few occasions. > > I would like to be able to write: > > range(100).\ > filter( f1 ).\ > map( f2 ).\ > filter( f3 ).\ > map( f4 ).\ > reduce(operator.add) > > in addition to current Pythonic way of > > reduce( operator.add, f4(x) for x in > ( f2(y) for y in range(100) if f1(y)) > if f3(x) ) > > Though longer - Java way seems to be a bit more readable as the notation > follows the data flow sequence. > > Do you think it is worth a PEP? > > BR, > > Jacek > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Apr 11 00:04:49 2014 From: barry at python.org (Barry Warsaw) Date: Thu, 10 Apr 2014 18:04:49 -0400 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. References: Message-ID: <20140410180449.7749600b@anarchist> Wouldn't another, much less invasive, much more explicit approach be to not squash subclasses of dict when passed into **args? IOW, the way to preserve order would be: >>> def foo(**kws): ... print(kws) ... >>> from collections import OrderedDict as od >>> foo(**od(a=1, b=2, c=3)) OrderedDict([('a', 1), ('c', 3), ('b', 2)]) >>> # instead of: {'a': 1, 'c': 3, 'b': 2} ? Okay, if you call it with keyword syntax, e.g. >>> foo(a=3, b=2, c=1) {'c': 1, 'a': 3, 'b': 2} but oh well. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From joshua at landau.ws Fri Apr 11 00:12:52 2014 From: joshua at landau.ws (Joshua Landau) Date: Thu, 10 Apr 2014 23:12:52 +0100 Subject: [Python-ideas] map, filter, reduce methods for generators In-Reply-To: References: Message-ID: On 10 April 2014 22:12, Jacek Pliszka wrote: > I would like to be able to write: > > range(100).\ > filter( f1 ).\ > map( f2 ).\ > filter( f3 ).\ > map( f4 ).\ > reduce(operator.add) > > in addition to current Pythonic way of > > reduce( operator.add, f4(x) for x in > ( f2(y) for y in range(100) if f1(y)) > if f3(x) ) The current Pythonic way is longname_1 = (f2(shortname) for shortname in range(100) if f1(shortname)) longname_2 = sum(f4(shortname) for shortname in longname_1 if f3(shortname)) If "shortname" and "longname_i" aren't to taste, find a context and assign good ones. Further, find me a plausibly-real-world example where my 2-line version is *less* readable than the Java one. Then I'll consider it a fair fight. From p.f.moore at gmail.com Fri Apr 11 00:16:00 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Apr 2014 23:16:00 +0100 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: <20140410180449.7749600b@anarchist> References: <20140410180449.7749600b@anarchist> Message-ID: On 10 April 2014 23:04, Barry Warsaw wrote: > Wouldn't another, much less invasive, much more explicit approach be to not > squash subclasses of dict when passed into **args? Doesn't this fail to support the only significant use case - keyword argument syntax for the OrderedDict constructor...? Paul From jacek.pliszka at gmail.com Fri Apr 11 00:09:22 2014 From: jacek.pliszka at gmail.com (Jacek Pliszka) Date: Fri, 11 Apr 2014 00:09:22 +0200 Subject: [Python-ideas] map, filter, reduce methods for generators In-Reply-To: References: Message-ID: OK, if method chaining is a problem, what about: from StreamAlgebra import Map, Filter, Reduce from operator import add range(100) | Filter(f1) | Map(f2) | Filter(f3) | Map(f4) | Reduce(add) BR, Jacek -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Apr 11 00:20:31 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Apr 2014 23:20:31 +0100 Subject: [Python-ideas] map, filter, reduce methods for generators In-Reply-To: References: Message-ID: On 10 April 2014 23:09, Jacek Pliszka wrote: > OK, if method chaining is a problem, what about: > > from StreamAlgebra import Map, Filter, Reduce > from operator import add > > range(100) | Filter(f1) | Map(f2) | Filter(f3) | Map(f4) | Reduce(add) It's not so much that these types of things are "a problem" - more that they are simply not common usage in Python, and so are unfamiliar. You can certainly write code that uses method chaining, or your StreamAlgebra module. They are entirely legitimate as 3rd party code. What's less clear is that they belong in the core / stdlib, because core code typically should be idiomatic, and these styles are not idiomatic Python (for better or worse). Paul From ericsnowcurrently at gmail.com Fri Apr 11 00:40:21 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 10 Apr 2014 16:40:21 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: <20140410180449.7749600b@anarchist> References: <20140410180449.7749600b@anarchist> Message-ID: On Thu, Apr 10, 2014 at 4:04 PM, Barry Warsaw wrote: > Wouldn't another, much less invasive, much more explicit approach be to not > squash subclasses of dict when passed into **args? > > IOW, the way to preserve order would be: > >>>> def foo(**kws): > ... print(kws) > ... >>>> from collections import OrderedDict as od >>>> foo(**od(a=1, b=2, c=3)) > OrderedDict([('a', 1), ('c', 3), ('b', 2)]) >>>> # instead of: {'a': 1, 'c': 3, 'b': 2} > > ? So preserve the type of the object passed in? Perhaps. However, using keyword args for OrderedDict loses order, so that would put us right back at the status quo anyway! > > Okay, if you call it with keyword syntax, e.g. > >>>> foo(a=3, b=2, c=1) > {'c': 1, 'a': 3, 'b': 2} Which is the main use case (rather than that of handling just unpacked args--which incidentally is a more complicated proposal). -eric From ericsnowcurrently at gmail.com Fri Apr 11 00:47:01 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 10 Apr 2014 16:47:01 -0600 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: <20140410180449.7749600b@anarchist> Message-ID: On Thu, Apr 10, 2014 at 4:16 PM, Paul Moore wrote: > On 10 April 2014 23:04, Barry Warsaw wrote: >> Wouldn't another, much less invasive, much more explicit approach be to not >> squash subclasses of dict when passed into **args? > > Doesn't this fail to support the only significant use case - keyword > argument syntax for the OrderedDict constructor...? While perhaps the first one people think of, I wouldn't say it's the only significant one. The PEP outlines several that are worth supporting better. While these aren't every-day annoyances, neither is this a proposal for new syntax. The only real downside is the performance on functions that have **kwargs in their signature, and even that has not been proven or disproven as a concern (I'm having computer problems so I haven't been able to get any concrete data). The last time I checked, most C OrderedDict operations were about the same as dict and the worst one took 4x as long. So just to make this clear, this proposal will only impact performance for calling a small subset of functions and I anticipate that impact will be small. -eric From abarnert at yahoo.com Fri Apr 11 05:02:09 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 10 Apr 2014 20:02:09 -0700 (PDT) Subject: [Python-ideas] map, filter, reduce methods for generators In-Reply-To: References: Message-ID: <1397185329.95019.YahooMailNeo@web181006.mail.ne1.yahoo.com> From: Jacek Pliszka Sent: Thursday, April 10, 2014 2:12 PM >What do you think about adding map, flatmap, filter and reduce methods to generator type ? That wouldn't help your intended use case, because range is not a generator.?In fact, most iterables are not generators. Non-iterators like list and dict, iterators defined as classes, iterators returned by builtins and C extension modules, etc. are not generators either.?So, do you want to somehow add this to all possible iterable types? Or do you want to force people to wrap an iterable inside an unnecessary generator (x for x in spam) just so they can call these methods on the wrapper? Or? ? And this isn't just a side issue; this gets to the heart of the difference between Python and Java. Java requires everything to be hammered into its OO paradigm. These are methods in Java because everything has to be a method in Java. In C++, Haskell, OCaml, or just about anything besides Java (and its cousins, like Ruby and various .NET languages), they're generic or polymorphic or duck-typed free functions that are defined once and work on any type that makes sense, instead of methods that have to be defined on?every possible type that might have a use for them. >I must admit I've seen and I like Java 8 notation and I think it might be more readable than Python way in a few occasions. > >I would like to be able to write: > >range(100).\ >? filter( f1 ).\ >? map( f2 ).\ >? filter( f3 ).\ >? map( f4 ).\ >? reduce(operator.add) > >in addition to current Pythonic way of > >reduce( operator.add, f4(x) for x in >? ( f2(y) for y in range(100) if f1(y)) >?if f3(x) ) This is not at all the Pythonic way to write it. And the fact that you think it is implies that maybe you're trying to solve a problem that doesn't exist. First, you're using reduce(add) instead of sum. I think this creates a false problem?if you think in terms of "map, filter, reduce", then?there's no way to get rid of some of the function calls piling up on the left. But if you really think about it, map and filter are different from reduce: they transform an iterable into an iterable, so you can call them any number of times in your sequence of transformations, but reduce transforms an iterable into a single value, so you only call it once. Which means there aren't function calls piling up on the left, there's exactly one function call on the left. Also, you're trying to cram everything into one expression for no good reason, which forces you to come up with some idiosyncratic way to wrap it to 80 columns. In Java, creating unnecessary temporary variables is often considered an anti-pattern, probably because of its C heritage (where it can be a performance issue). In Python, this is instead a very common idiom. Let's start with the simplest possible way to write this: ? ? r = range(100) ? ? r = filter(f1, r) ? ? r = map(f2, r) ? ? r = filter(f3, r) ? ? r = map(f4, r) Now, taking advantage of comprehensions: ? ? r = range(100) ? ? r = (f2(x) for x in r if f1(x)) ? ? r = (f4(x) for x in r if f3(x)) This may look like the syntax is hiding the real functionality, but that's only because the real functionality is invisible in your example, because you've named the functions f1, f2, f3, and f4. Try an example with realistic function names and it will look a lot different.?And of course half the time, you don't actually have a function lying around, you just want to map or filter with some expression, in which case the sequence of comprehensions wins even bigger. Compare: ? ? found_squares = (x**2 for x in range(100) if x in found) ? ? weighted_sum = sum(x / dups[x] for x in r if x in found_dups) ? ? weighted_sum = range(100). \ ? ? ? ? filter(lambda x: x in found). \ # or found.__contains__ if you insist ? ? ? ? map(lambda x: x**2). \ ? ? ? ? filter(lambda x: x in found_dups). \ ? ? ? ? map(lambda x: x / dups[x]). ? ? ? ? reduce(operator.add) You really think the second one is more readable? Notice that in the first one, everything is happening in the order of the data flow, you're not piling up function calls on the left, etc.; all the advantages you're looking for.?If you haven't read David Beazley's "Generator Tricks for System Programmers", google it for some great realistic examples (and some great background discussion, too). From abarnert at yahoo.com Fri Apr 11 05:20:30 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 10 Apr 2014 20:20:30 -0700 (PDT) Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: <20140410180449.7749600b@anarchist> References: <20140410180449.7749600b@anarchist> Message-ID: <1397186430.32047.YahooMailNeo@web181004.mail.ne1.yahoo.com> From: Barry Warsaw Sent: Thursday, April 10, 2014 3:04 PM > Wouldn't another, much less invasive, much more explicit approach be to not > squash subclasses of dict when passed into **args? > > IOW, the way to preserve order would be: > >>>> def foo(**kws): > ...? print(kws) > ... >>>> from collections import OrderedDict as od >>>> foo(**od(a=1, b=2, c=3)) > OrderedDict([('a', 1), ('c', 3), ('b', 2)]) >>>> # instead of: {'a': 1, 'c': 3, 'b': 2} I think your own example shows why this doesn't work. You wanted to pass in a=1, b=2, c=3 by passing it through an OrderedDict? but you ended up passing a=1, c=3, b=2 instead. So you've successfully preserved _an_ order in kws, but not the one you wanted. (Also, a **kws parameter doesn't get a squashed copy of the **d argument in the first place; it gets a new dict, into which things get copied. And the things that get copied aren't the members of d unless?both exist, and there are no members of d that match named parameters, and there are no keyword arguments that don't match named parameters.?So, I don't think this suggestion is even a coherent solution, much less a successful one. But you could obviously fix that by saying that, e.g., you create a new type(d) instead of a new dict, and then add things to that normally.) From jbvsmo at gmail.com Fri Apr 11 12:55:35 2014 From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=) Date: Fri, 11 Apr 2014 07:55:35 -0300 Subject: [Python-ideas] map, filter, reduce methods for generators In-Reply-To: References: Message-ID: Some guy did a similar thing, just better: https://github.com/JulienPalard/Pipe BTW, Changing the built-in iterators would require a change in the abc's. Jo?o Bernardo On 10 April 2014 19:09, Jacek Pliszka wrote: > OK, if method chaining is a problem, what about: > > from StreamAlgebra import Map, Filter, Reduce > from operator import add > > range(100) | Filter(f1) | Map(f2) | Filter(f3) | Map(f4) | Reduce(add) > > BR, > > Jacek > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Apr 11 12:58:56 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 11 Apr 2014 06:58:56 -0400 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: <1397186430.32047.YahooMailNeo@web181004.mail.ne1.yahoo.com> References: <20140410180449.7749600b@anarchist> <1397186430.32047.YahooMailNeo@web181004.mail.ne1.yahoo.com> Message-ID: <20140411065856.1589ba5f@anarchist> On Apr 10, 2014, at 08:20 PM, Andrew Barnert wrote: >I think your own example shows why this doesn't work. Not really. It was just a silly example for conciseness. I don't care how you build up the OrderedDict, and there are several order-preserving ways to do that. I'm not sure it's a good idea to change *all of Python* just to make the OD constructor nicer. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From ncoghlan at gmail.com Fri Apr 11 13:49:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 Apr 2014 07:49:15 -0400 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: <20140411065856.1589ba5f@anarchist> References: <20140410180449.7749600b@anarchist> <1397186430.32047.YahooMailNeo@web181004.mail.ne1.yahoo.com> <20140411065856.1589ba5f@anarchist> Message-ID: On 11 Apr 2014 07:00, "Barry Warsaw" wrote: > > On Apr 10, 2014, at 08:20 PM, Andrew Barnert wrote: > > >I think your own example shows why this doesn't work. > > Not really. It was just a silly example for conciseness. I don't care how > you build up the OrderedDict, and there are several order-preserving ways to > do that. I'm not sure it's a good idea to change *all of Python* just to make > the OD constructor nicer. It's not "all of Python" - it's only functions that accept **kwargs. And when your code is speed critical, doing that's already a bad idea. Cheers, Nick. > > -Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Apr 11 14:07:04 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 11 Apr 2014 08:07:04 -0400 Subject: [Python-ideas] Proto-PEP: Preserving the order of **kwargs in a function. In-Reply-To: References: <20140410180449.7749600b@anarchist> <1397186430.32047.YahooMailNeo@web181004.mail.ne1.yahoo.com> <20140411065856.1589ba5f@anarchist> Message-ID: On Fri, Apr 11, 2014 at 7:49 AM, Nick Coghlan wrote: > > Not really. It was just a silly example for conciseness. I don't care > how > > you build up the OrderedDict, and there are several order-preserving > ways to > > do that. I'm not sure it's a good idea to change *all of Python* just > to make > > the OD constructor nicer. > > It's not "all of Python" - it's only functions that accept **kwargs. And > when your code is speed critical, doing that's already a bad idea. > And it is not just about OrderedDict. There are many other examples where it is helpful to be able to supply arbitrary keyword arguments and have the order preserved. For example, when building a memory view to a C struct, a nice interface would be struct_view(foo=int, bar=float). Another example is constructing tables. One may want to supply column info in keyword arguments named as columns. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Apr 11 21:03:21 2014 From: brett at python.org (Brett Cannon) Date: Fri, 11 Apr 2014 15:03:21 -0400 Subject: [Python-ideas] [RFC] Draft-PEP: Adding structured data to built-in exceptions In-Reply-To: References: Message-ID: On Sun, Apr 6, 2014 at 1:32 PM, Sebastian Kreft wrote: > Continuing the discussion started in > https://mail.python.org/pipermail/python-ideas/2014-February/025601.html, > attached is a draft PEP explaining the rationale of the idea and the > required changes. > > I'm looking forward to your comments. > I currently don't have time to read the PEP, but I didn't notice references to bugs I previously on this exact topic: http://bugs.python.org/issue18162 http://bugs.python.org/issue18156 http://bugs.python.org/issue18166 http://bugs.python.org/issue18165 http://bugs.python.org/issue18163 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Fri Apr 11 21:37:02 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 11 Apr 2014 12:37:02 -0700 Subject: [Python-ideas] [RFC] Draft-PEP: Adding structured data to built-in exceptions In-Reply-To: References: Message-ID: I think you need to make clear when exceptions will have these attributes. Is it always, whenever the exception is raised by Python itself or anything in the Python standard library, and otherwise depends on the raiser? P.S. +1 because https://mail.python.org/pipermail/python-ideas/2013-April/020308.html ;) -- Devin On Sun, Apr 6, 2014 at 10:32 AM, Sebastian Kreft wrote: > Continuing the discussion started in > https://mail.python.org/pipermail/python-ideas/2014-February/025601.html, > attached is a draft PEP explaining the rationale of the idea and the > required changes. > > I'm looking forward to your comments. > > -- > Sebastian Kreft > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From daoust.mj at gmail.com Sat Apr 12 01:23:14 2014 From: daoust.mj at gmail.com (Mark Daoust) Date: Fri, 11 Apr 2014 19:23:14 -0400 Subject: [Python-ideas] map, filter, reduce methods for generators In-Reply-To: References: Message-ID: On Fri, Apr 11, 2014 at 6:55 AM, Jo?o Bernardo wrote: > Some guy did a similar thing, just better: > https://github.com/JulienPalard/Pipe > > Jo?o Bernardo > > It's interesting how that one puts the logic in the pipe elements instead of the iterators. Here's one that does the method chaining route, kind of an "Itertools: the class". http://code.activestate.com/recipes/498272-rich-iterator-wrapper/ I like that connects indexing to `islice`. Mark Daoust -------------- next part -------------- An HTML attachment was scrubbed... URL: From retrobanana.jn at gmail.com Sat Apr 12 19:14:18 2014 From: retrobanana.jn at gmail.com (James Nelson) Date: Sat, 12 Apr 2014 11:14:18 -0600 Subject: [Python-ideas] No implicit variable declarations Message-ID: There are several problems with Python's implicit variable declaration. Just one tiny mistake, like assigning to SpamEggs instead of spamEggs will create a new variable SpamEggs, and spamEggs will hold a stale value. Two statements should be added to Python: (1) a statement that specifies that implicit declaration by assignment should be disabled, like Fortran's "implicit none", and (2) a variable declaration statement, a la "var spamEggs". Trying to assign to a non-existent variable would, of course, cause an error. If the non-existent variable's name is close enough to an existing variable's name, the interpreter might suggest the correct name: "Did you mean "spamEggs"?". This idea could even open Python to static typed variables, declared like "var spamEggs as int". This would prevent the stupid but very possible error of assigning "clientName" (a string) to "doorStatus" (a boolean). -rb -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Apr 12 19:23:39 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 12 Apr 2014 10:23:39 -0700 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: Message-ID: <5349769B.6080902@stoneleaf.us> On 04/12/2014 10:14 AM, James Nelson wrote: > There are several problems with Python's implicit variable declaration. Just one tiny mistake, like assigning to > SpamEggs instead of spamEggs will create a new variable SpamEggs, and spamEggs will hold a stale value. This is why you should have unit tests. > Two statements should be added to Python: (1) a statement that specifies that implicit declaration by assignment should > be disabled, like Fortran's "implicit none", and (2) a variable declaration statement, a la "var spamEggs". Trying to > assign to a non-existent variable would, of course, cause an error. If the non-existent variable's name is close enough > to an existing variable's name, the interpreter might suggest the correct name: "Did you mean "spamEggs"?". And also linters [1]. > This idea could even open Python to static typed variables, declared like "var spamEggs as int". This would prevent the > stupid but very possible error of assigning "clientName" (a string) to "doorStatus" (a boolean). -1 on the whole thing. -- ~Ethan~ [1] http://stackoverflow.com/q/5611776/208880 From markus at unterwaditzer.net Sat Apr 12 19:47:42 2014 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Sat, 12 Apr 2014 19:47:42 +0200 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: Message-ID: <20140412174742.GA10254@untibox.unti> Hello James, I see how this would help with avoiding a whole class of errors, but then i also think your two suggestions -- the var and implicit keywords -- would take a great deal of flexibility and expressiveness from the language. Also, Python was never meant to be statically typed, so suggesting static typing so many years after its creation is IMO pointless, even though you're suggesting it to be optional. OTOH I like your idea of suggesting variable names when NameError is raised, but i don't know how easy this is to implement in CPython. I also want to say that the kind of error you're describing never occured to me in practice. The mistake of assigning to the wrong variable name (and not noticing it immediately through an exception) seems to me like something that only occurs when deliberately writing huge cludges of spaghetti-code, or when you're simply not using enough namespaces. -- Markus On Sat, Apr 12, 2014 at 11:14:18AM -0600, James Nelson wrote: > There are several problems with Python's implicit variable declaration. > Just one tiny mistake, like assigning to SpamEggs instead of spamEggs will > create a new variable SpamEggs, and spamEggs will hold a stale value. > > Two statements should be added to Python: (1) a statement that specifies > that implicit declaration by assignment should be disabled, like Fortran's > "implicit none", and (2) a variable declaration statement, a la "var > spamEggs". Trying to assign to a non-existent variable would, of course, > cause an error. If the non-existent variable's name is close enough to an > existing variable's name, the interpreter might suggest the correct name: > "Did you mean "spamEggs"?". > > This idea could even open Python to static typed variables, declared like > "var spamEggs as int". This would prevent the stupid but very possible > error of assigning "clientName" (a string) to "doorStatus" (a boolean). > > -rb > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rymg19 at gmail.com Sat Apr 12 20:06:01 2014 From: rymg19 at gmail.com (Ryan) Date: Sat, 12 Apr 2014 13:06:01 -0500 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: Message-ID: <7721be8a-234a-4fd1-bf96-335dc0cbee20@email.android.com> Isn't this what MyPy is for? James Nelson wrote: >There are several problems with Python's implicit variable declaration. >Just one tiny mistake, like assigning to SpamEggs instead of spamEggs >will >create a new variable SpamEggs, and spamEggs will hold a stale value. > >Two statements should be added to Python: (1) a statement that >specifies >that implicit declaration by assignment should be disabled, like >Fortran's >"implicit none", and (2) a variable declaration statement, a la "var >spamEggs". Trying to assign to a non-existent variable would, of >course, >cause an error. If the non-existent variable's name is close enough to >an >existing variable's name, the interpreter might suggest the correct >name: >"Did you mean "spamEggs"?". > >This idea could even open Python to static typed variables, declared >like >"var spamEggs as int". This would prevent the stupid but very possible >error of assigning "clientName" (a string) to "doorStatus" (a boolean). > >-rb > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >https://mail.python.org/mailman/listinfo/python-ideas >Code of Conduct: http://python.org/psf/codeofconduct/ -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 12 20:07:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Apr 2014 04:07:53 +1000 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: <20140412174742.GA10254@untibox.unti> References: <20140412174742.GA10254@untibox.unti> Message-ID: On Sun, Apr 13, 2014 at 3:47 AM, Markus Unterwaditzer wrote: > OTOH I like your idea of suggesting variable names when NameError > is raised, but i don't know how easy this is to implement in CPython. There's another proposal in the pipeline, regarding the addition of extra info to various exceptions, and one of the points it makes is that suggestions could be made on NameError (which would probably be a feature wanted more in interactive mode than elsewhere). How easily it can be made truly useful is anyone's guess; I generally find that "Did you mean" lists are more often useless than helpful. ChrisA From retrobanana.jn at gmail.com Sat Apr 12 20:14:04 2014 From: retrobanana.jn at gmail.com (James Nelson) Date: Sat, 12 Apr 2014 18:14:04 +0000 (UTC) Subject: [Python-ideas] No implicit variable declarations References: <20140412174742.GA10254@untibox.unti> Message-ID: Markus Unterwaditzer writes: > > Hello James, > > I see how this would help with avoiding a whole class of errors, but then i > also think your two suggestions -- the var and implicit keywords -- would take > a great deal of flexibility and expressiveness from the language. Also, Python > was never meant to be statically typed, so suggesting static typing so many > years after its creation is IMO pointless, even though you're suggesting it to > be optional. OTOH I like your idea of suggesting variable names when NameError > is raised, but i don't know how easy this is to implement in CPython. > > I also want to say that the kind of error you're describing never occured to me > in practice. The mistake of assigning to the wrong variable name (and not > noticing it immediately through an exception) seems to me like something that > only occurs when deliberately writing huge cludges of spaghetti-code, or when > you're simply not using enough namespaces. > I don't believe that having var and implicit keywords would make the language less flexible -- they are completely optional, remember. If you omit "implicit none" from the beginning of your program you can write your program like you do now, implicit variable declarations and all. In retrospect, statically typed variables does contradict Python's duck typing: it shouldn't matter if it's a float or an integer, as long as you multiply or divide, you're fine, and if you need a specific type, you just check with type(). - JN From markus at unterwaditzer.net Sat Apr 12 20:53:40 2014 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Sat, 12 Apr 2014 20:53:40 +0200 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: <20140412174742.GA10254@untibox.unti> Message-ID: <20140412185340.GA20841@untibox.unti> On Sat, Apr 12, 2014 at 06:14:04PM +0000, James Nelson wrote: > I don't believe that having var and implicit keywords would make the > language less flexible -- they are completely optional, remember. If you > omit "implicit none" from the beginning of your program you can write your > program like you do now, implicit variable declarations and all. At which scope should this statement operate? Should it only be allowed at the top of a module (like __future__ imports), or should it operate on a per-function basis? Doing this for the whole process, as i think your usage of the term "program" implies, sounds like a very bad idea to me. Anyway, i think that this idea just doesn't "fit" Python, and "implicit none" kind-of reminds me of the strict mode in Javascript. But other than in Javascript, i don't think Python actually *needs* this. As said, linters cover these problems for me. -- Markus From rosuav at gmail.com Sat Apr 12 21:02:06 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Apr 2014 05:02:06 +1000 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: <20140412185340.GA20841@untibox.unti> References: <20140412174742.GA10254@untibox.unti> <20140412185340.GA20841@untibox.unti> Message-ID: On Sun, Apr 13, 2014 at 4:53 AM, Markus Unterwaditzer wrote: > On Sat, Apr 12, 2014 at 06:14:04PM +0000, James Nelson wrote: >> I don't believe that having var and implicit keywords would make the >> language less flexible -- they are completely optional, remember. If you >> omit "implicit none" from the beginning of your program you can write your >> program like you do now, implicit variable declarations and all. > > At which scope should this statement operate? Should it only be allowed at the > top of a module (like __future__ imports), or should it operate on a > per-function basis? Doing this for the whole process, as i think your usage of > the term "program" implies, sounds like a very bad idea to me. As it's something that doesn't change bytecode execution (other than the extra info on NameError, which as mentioned can be done regardless of this proposal) but effects a change to syntax, I would recommend it be per-module, like a future directive. ChrisA From joseph.martinot-lagarde at m4x.org Sat Apr 12 21:28:40 2014 From: joseph.martinot-lagarde at m4x.org (Joseph Martinot-Lagarde) Date: Sat, 12 Apr 2014 21:28:40 +0200 Subject: [Python-ideas] keyword-only args in __getitem__ Message-ID: I propose to allow to define keyword-only arguments for __getitem__. The main use case is to be able to set a default value for indexing. I though about it after reading PEP 463 (Exception-catching expressions), this would be an alternative for one of the problem this PEP is trying to solve. Compare these solutions: >>> lst = [1, 2, 3] >>> value = lst[3] Traceback (most recent call last): File "", line 1, in IndexError: list index out of range >>> try: # Python .... value = lst[3] .... except IndexError: .... value = 0 >>> value = lst[3] except IndexError: 0 # PEP 463 >>> value = lst[3, default=0] # My proposal Default value is not the only use case. One could for example define a subclass of list computing an interpolation if a float is given as a parameter instead of truncating it, and the keword argument would define which kind of interpolation (nearest, linear, spline...). Or it could define if the value returned are in degrees or radians. (Yes, I have a scientific background.) I use something similar for a dictionary-like interface where the keys are strings and only numbers are stored. The second argument, if given, is the default: >>> data["key", 0] It would be clearer to use: >>> data["key", default=0] (Of course I could use a get method for this purpose, but bracket for indexing seems always more clear). The keyword would be keywords-only because right now when many arguments are passed to __getitem__, they are grouped in a tuple and this tuple is passed to __getitem__ as a unique argument. Also it is more clear: ordered arguments are the indexes, keyword arguments are the options. To be perfectly honest I don't think that this proposal is that great, it's main selling point is that a "default" keyword in indexing would be a good alternative for some of the problems raised in PEP 463. Joseph ML --- Ce courrier ?lectronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com From bruce at leapyear.org Sat Apr 12 21:44:46 2014 From: bruce at leapyear.org (Bruce Leban) Date: Sat, 12 Apr 2014 12:44:46 -0700 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: Message-ID: I think a much more productive approach would be to think about improving pylint by adding support for type comments. Look at how Google's closure compiler does this for JavaScript. --- Bruce (from my phone) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Apr 13 00:30:47 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 12 Apr 2014 18:30:47 -0400 Subject: [Python-ideas] keyword-only args in __getitem__ In-Reply-To: References: Message-ID: On 4/12/2014 3:28 PM, Joseph Martinot-Lagarde wrote: > I propose to allow to define keyword-only arguments for __getitem__. > > The main use case is to be able to set a default value for indexing. I > though about it after reading PEP 463 (Exception-catching expressions), > this would be an alternative for one of the problem this PEP is trying > to solve. > Compare these solutions: > >>> lst = [1, 2, 3] > >>> value = lst[3] > Traceback (most recent call last): > File "", line 1, in > IndexError: list index out of range > >>> try: # Python > .... value = lst[3] > .... except IndexError: > .... value = 0 > >>> value = lst[3] except IndexError: 0 # PEP 463 > >>> value = lst[3, default=0] # My proposal The problem with this is, as you note below, that everything between [ and ] is turned into a single object passed on to __getitem__. That includes slice notation I:J:I, which is currently only valid in this context. Syntactically, collection indexing is quite different from function calling. (Python could have been defined so that they were the same.) I think it would be better if the key were syntactically separated from a default, say with ;. value = lst[3; 0] I believe any proposal along this line would make dict.getitem obsolete, and eliminate any call for list.getitem. > Default value is not the only use case. One could for example define a > subclass of list computing an interpolation if a float is given as a > parameter instead of truncating it, and the keword argument would define > which kind of interpolation (nearest, linear, spline...). I think interp should be a method of the subclass. > Or it could > define if the value returned are in degrees or radians. (Yes, I have a > scientific background.) .setunit('degrees') with radians the default. > I use something similar for a dictionary-like interface where the keys > are strings and only numbers are stored. The second argument, if given, > is the default: > >>> data["key", 0] > It would be clearer to use: > >>> data["key", default=0] I see data['key'; 0] as an in between compromise. The 0 is separated from the key but without the heavy 'default='. > (Of course I could use a get method for this purpose, but bracket for > indexing seems always more clear). -- Terry Jan Reedy From rosuav at gmail.com Sun Apr 13 02:45:22 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Apr 2014 10:45:22 +1000 Subject: [Python-ideas] keyword-only args in __getitem__ In-Reply-To: References: Message-ID: On Sun, Apr 13, 2014 at 5:28 AM, Joseph Martinot-Lagarde wrote: > The main use case is to be able to set a default value for indexing. I > though about it after reading PEP 463 (Exception-catching expressions), this > would be an alternative for one of the problem this PEP is trying to solve. > Compare these solutions: >>>> lst = [1, 2, 3] >>>> value = lst[3] > Traceback (most recent call last): > File "", line 1, in > IndexError: list index out of range >>>> try: # Python > .... value = lst[3] > .... except IndexError: > .... value = 0 >>>> value = lst[3] except IndexError: 0 # PEP 463 >>>> value = lst[3, default=0] # My proposal An interesting idea, but that looks far too much like a function call. Square brackets and then more and more a function call inside... may fail the grit test. Part of the point of except expressions was that the implementation of __getitem__ wouldn't need to have two branches in it. Currently, you need to write __getitem__ (which raises an exception on finding a problem) plus something else, eg get(), which returns a default instead. By your proposal, both branches would go inside __getitem__, which means they could share code; but there still need to be two branches. That's two branches that need to be written, tested, etc, and if the author hasn't thought to handle default=, there's no way to get that functionality, same as if there's no get() method. And just as with the get method, there'll be an ad-hoc and fairly arbitrary puddle of names (some will go "default=", others will say that's way too long and go "def=", except that that's a keyword so they'll use "dflt=" or something...), unless there's a strong force pushing people to one consistent name. Terry's suggestion solves that part of it, but also restricts the feature to just default values; with a more general keyword argument system, it would make sense to also use this with __setitem__ and __delitem__, and I'm not sure how "default value" should be interpreted there. Which is better is, of course, a matter of opinion! :) ChrisA From abarnert at yahoo.com Sun Apr 13 06:21:03 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 12 Apr 2014 21:21:03 -0700 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: <20140412174742.GA10254@untibox.unti> Message-ID: On Apr 12, 2014, at 11:14, James Nelson wrote: > In retrospect, statically typed variables does contradict Python's duck > typing: it shouldn't matter if it's a float or an integer, Or, more importantly, some type you never thought of--maybe it's a numpy array of int16s... > as long as you > multiply or divide, you're fine, and if you need a specific type, you just > check with type(). Actually, even if you need a specific type, you usually want to check with isinstance. Only use type if you explicitly don't want to handle subclasses, ADT registration, etc., which is very rare. Anyway, while you presented this as an afterthought, duck typing is the whole point of dynamic languages like Python. You get the flexibility of a powerful polymorphic type system (like the ones in ML or Haskell, or to a lesser extent C++), while only having to ever think about (and write) simple types (like the ones in pre-generics Java or ActionScript). Of course the cost is that you don't get compile-time type checking. But for many programs, that type checking is nearly useless (at least the Java kind; the Haskell kind is more useful). Anyway, Python has had annotations for years now, and one of the reasons they were added was so someone could write a type-checking linter that statically checked optionally-annotated function and parameter types, and nobody is using it for that. Which implies that nobody is missing it too badly. From rosuav at gmail.com Sun Apr 13 09:56:18 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 13 Apr 2014 17:56:18 +1000 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: <20140412174742.GA10254@untibox.unti> Message-ID: On Sun, Apr 13, 2014 at 2:21 PM, Andrew Barnert wrote: > Of course the cost is that you don't get compile-time type checking. But for many programs, that type checking is nearly useless (at least the Java kind; the Haskell kind is more useful). > Aside: I have periodically had bugs where I omit the first argument of a function (in situations where, had it not been first, it would have been marked optional). In C, those sorts of bugs are often caught by the function signature; in Python, they can be, but only if the function takes a fixed number of args. For instance: void foo(target *destination, const char *msg, int blah=0, int blahblah=0) If you omit the destination, you get an error back straight away, because a string (const char *) can't be converted into a target. The equivalent in Python, though: def foo(destination, msg, blah=0, blahblah=0): would happily assign the args one slot earlier, as long as I provided at least one of the optionals. But this is a fairly narrow case; if both those args were mandatory, the mere count of arguments would trip an error. (Albeit at run-time, not compile-time, but until Python goes to actually run the line of code, it can't know that this is the signature anyway.) ChrisA PS. Neat trick with dmarc. From ncoghlan at gmail.com Sun Apr 13 11:32:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Apr 2014 05:32:37 -0400 Subject: [Python-ideas] keyword-only args in __getitem__ In-Reply-To: References: Message-ID: On 12 Apr 2014 20:54, "Chris Angelico" wrote: > > On Sun, Apr 13, 2014 at 5:28 AM, Joseph Martinot-Lagarde > wrote: > > The main use case is to be able to set a default value for indexing. I > > though about it after reading PEP 463 (Exception-catching expressions), this > > would be an alternative for one of the problem this PEP is trying to solve. > > Compare these solutions: > >>>> lst = [1, 2, 3] > >>>> value = lst[3] > > Traceback (most recent call last): > > File "", line 1, in > > IndexError: list index out of range > >>>> try: # Python > > .... value = lst[3] > > .... except IndexError: > > .... value = 0 > >>>> value = lst[3] except IndexError: 0 # PEP 463 > >>>> value = lst[3, default=0] # My proposal > > An interesting idea, but that looks far too much like a function call. > Square brackets and then more and more a function call inside... may > fail the grit test. > > Part of the point of except expressions was that the implementation of > __getitem__ wouldn't need to have two branches in it. Currently, you > need to write __getitem__ (which raises an exception on finding a > problem) plus something else, eg get(), which returns a default > instead. By your proposal, both branches would go inside __getitem__, > which means they could share code; but there still need to be two > branches. That's two branches that need to be written, tested, etc, > and if the author hasn't thought to handle default=, there's no way to > get that functionality, same as if there's no get() method. And just > as with the get method, there'll be an ad-hoc and fairly arbitrary > puddle of names (some will go "default=", others will say that's way > too long and go "def=", except that that's a keyword so they'll use > "dflt=" or something...), unless there's a strong force pushing people > to one consistent name. Terry's suggestion solves that part of it, but > also restricts the feature to just default values; with a more general > keyword argument system, it would make sense to also use this with > __setitem__ and __delitem__, and I'm not sure how "default value" > should be interpreted there. Which is better is, of course, a matter > of opinion! :) A getitem() builtin seems like a more obvious parallel with getattr(), and pairs up nicely with operator.itemgetter. Proposals of additional special cases like this is a reasonable outcome of the rejection of the except expressions PEP. Cheers, Nick. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sun Apr 13 19:21:28 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sun, 13 Apr 2014 13:21:28 -0400 Subject: [Python-ideas] keyword-only args in __getitem__ In-Reply-To: References: Message-ID: On 04/12/2014 06:30 PM, Terry Reedy wrote: > On 4/12/2014 3:28 PM, Joseph Martinot-Lagarde wrote: >> I propose to allow to define keyword-only arguments for __getitem__. >> >> The main use case is to be able to set a default value for indexing. I >> though about it after reading PEP 463 (Exception-catching expressions), >> this would be an alternative for one of the problem this PEP is trying >> to solve. >> Compare these solutions: >> >>> lst = [1, 2, 3] >> >>> value = lst[3] >> Traceback (most recent call last): >> File "", line 1, in >> IndexError: list index out of range >> >>> try: # Python >> .... value = lst[3] >> .... except IndexError: >> .... value = 0 >> >>> value = lst[3] except IndexError: 0 # PEP 463 >> >>> value = lst[3, default=0] # My proposal > > The problem with this is, as you note below, that everything between [ and > ] is turned into a single object passed on to __getitem__. The comma in slice notation does support passing multiple objects to get item. It's used in numpy for multidimensional lookups. I believe there is interest for adding that feature to python, so it may cause an issue with those proposals if the comma is used. But it seems to me a better way to do this may be with default method that returns a view or wrapped object that supplies a default rather than raise an index or key error. lst.default_view(0)[3] Generally you won't use different defaults with the same object that often, so it makes sense to set it early, and then not to have to do anything special in other places the object is used. Cheers, Ron From joseph.martinot-lagarde at m4x.org Sun Apr 13 22:21:14 2014 From: joseph.martinot-lagarde at m4x.org (Joseph Martinot-Lagarde) Date: Sun, 13 Apr 2014 22:21:14 +0200 Subject: [Python-ideas] keyword-only args in __getitem__ In-Reply-To: References: Message-ID: <534AF1BA.8050007@m4x.org> Le 13/04/2014 19:21, Ron Adam a ?crit : > > > On 04/12/2014 06:30 PM, Terry Reedy wrote: >> On 4/12/2014 3:28 PM, Joseph Martinot-Lagarde wrote: >>> I propose to allow to define keyword-only arguments for __getitem__. >>> >>> The main use case is to be able to set a default value for indexing. I >>> though about it after reading PEP 463 (Exception-catching expressions), >>> this would be an alternative for one of the problem this PEP is trying >>> to solve. >>> Compare these solutions: >>> >>> lst = [1, 2, 3] >>> >>> value = lst[3] >>> Traceback (most recent call last): >>> File "", line 1, in >>> IndexError: list index out of range >>> >>> try: # Python >>> .... value = lst[3] >>> .... except IndexError: >>> .... value = 0 >>> >>> value = lst[3] except IndexError: 0 # PEP 463 >>> >>> value = lst[3, default=0] # My proposal >> >> The problem with this is, as you note below, that everything between [ >> and >> ] is turned into a single object passed on to __getitem__. > > The comma in slice notation does support passing multiple objects to get > item. It's used in numpy for multidimensional lookups. I believe there > is interest for adding that feature to python, so it may cause an issue > with those proposals if the comma is used. > > > But it seems to me a better way to do this may be with default method > that returns a view or wrapped object that supplies a default rather > than raise an index or key error. > > lst.default_view(0)[3] > > Generally you won't use different defaults with the same object that > often, so it makes sense to set it early, and then not to have to do > anything special in other places the object is used. It may be true for lists as they are supposed to hold objects of the same kind. But for dictionnaries, you may want to have different defaults for each key. For example a dictionary containing the keys 'START', 'STOP' and 'STEP' would have something like 0, None and 1 as defaults. > > > Cheers, > Ron > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > --- Ce courrier ?lectronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com From joseph.martinot-lagarde at m4x.org Sun Apr 13 22:32:57 2014 From: joseph.martinot-lagarde at m4x.org (Joseph Martinot-Lagarde) Date: Sun, 13 Apr 2014 22:32:57 +0200 Subject: [Python-ideas] keyword-only args in __getitem__ In-Reply-To: References: Message-ID: <534AF479.4030805@m4x.org> Le 13/04/2014 02:45, Chris Angelico a ?crit : > On Sun, Apr 13, 2014 at 5:28 AM, Joseph Martinot-Lagarde > wrote: >> The main use case is to be able to set a default value for indexing. I >> though about it after reading PEP 463 (Exception-catching expressions), this >> would be an alternative for one of the problem this PEP is trying to solve. >> Compare these solutions: >>>>> lst = [1, 2, 3] >>>>> value = lst[3] >> Traceback (most recent call last): >> File "", line 1, in >> IndexError: list index out of range >>>>> try: # Python >> .... value = lst[3] >> .... except IndexError: >> .... value = 0 >>>>> value = lst[3] except IndexError: 0 # PEP 463 >>>>> value = lst[3, default=0] # My proposal > > An interesting idea, but that looks far too much like a function call. > Square brackets and then more and more a function call inside... may > fail the grit test. I tend to see square brackets as a function call but "indexing oriented". Using parenthesis would correspond to calling __call__ while using brackets corresponds to calling __getitem__. And sometimes I'd like to be able to use I:J:K notation for slices in function calls... > > Part of the point of except expressions was that the implementation of > __getitem__ wouldn't need to have two branches in it. Currently, you > need to write __getitem__ (which raises an exception on finding a > problem) plus something else, eg get(), which returns a default > instead. By your proposal, both branches would go inside __getitem__, > which means they could share code; but there still need to be two > branches. That's two branches that need to be written, tested, etc, > and if the author hasn't thought to handle default=, there's no way to > get that functionality, same as if there's no get() method. Yes the code would be included by the author rather than by the user. Part of the success of python is that it comes with batteries included ! > And just > as with the get method, there'll be an ad-hoc and fairly arbitrary > puddle of names (some will go "default=", others will say that's way > too long and go "def=", except that that's a keyword so they'll use > "dflt=" or something...), unless there's a strong force pushing people > to one consistent name. Well, this argument would hold against using keywords arguments in any function call. > Terry's suggestion solves that part of it, but > also restricts the feature to just default values; with a more general > keyword argument system, it would make sense to also use this with > __setitem__ and __delitem__, and I'm not sure how "default value" > should be interpreted there. Which is better is, of course, a matter > of opinion! :) setitem and delitem don't look like function calls at all IMHO. Keywords arguments for them would be less useful than for getitem. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > --- Ce courrier ?lectronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active. http://www.avast.com From rymg19 at gmail.com Mon Apr 14 01:22:52 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 13 Apr 2014 18:22:52 -0500 Subject: [Python-ideas] No implicit variable declarations In-Reply-To: References: <20140412174742.GA10254@untibox.unti> Message-ID: On Sun, Apr 13, 2014 at 2:56 AM, Chris Angelico wrote: > On Sun, Apr 13, 2014 at 2:21 PM, Andrew Barnert > wrote: > > Of course the cost is that you don't get compile-time type checking. But > for many programs, that type checking is nearly useless (at least the Java > kind; the Haskell kind is more useful). > > > > Aside: I have periodically had bugs where I omit the first argument of > a function (in situations where, had it not been first, it would have > been marked optional). In C, those sorts of bugs are often caught by > the function signature; in Python, they can be, but only if the > function takes a fixed number of args. For instance: > > void foo(target *destination, const char *msg, int blah=0, int blahblah=0) > > If you omit the destination, you get an error back straight away, > because a string (const char *) can't be converted into a target. The > equivalent in Python, though: > No you don't! If you're stuck in C89 for one reason or another, all you get is a warning(which most programmers ignore anyway). C++ is much more strict in that regard. > def foo(destination, msg, blah=0, blahblah=0): > > would happily assign the args one slot earlier, as long as I provided > at least one of the optionals. But this is a fairly narrow case; if > both those args were mandatory, the mere count of arguments would trip > an error. (Albeit at run-time, not compile-time, but until Python goes > to actually run the line of code, it can't know that this is the > signature anyway.) > > ChrisA > > PS. Neat trick with dmarc. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From dev at joernhees.de Mon Apr 14 17:51:28 2014 From: dev at joernhees.de (=?iso-8859-1?Q?J=F6rn_Hees?=) Date: Mon, 14 Apr 2014 17:51:28 +0200 Subject: [Python-ideas] module version number support for semver.org Message-ID: Hi, what do you think about officially supporting Semantic Versioning? ( http://semver.org ) Semantic Versioning has gained a lot of attention in other programming languages. Even though not officially supported, many python libraries use it as well (just search the pypi listing for things like "-alpha" "-beta" "-dev" "-pre" to get a rough idea). There even is a class handling Semantic Versioning in pip already: pip.util.version.SemanticVersion . I'd speak in favor of officially adding support for semantic versioning to the python module versioning specs as a follow up to PEP 440 ( http://legacy.python.org/dev/peps/pep-0440/ ). I want to avoid the fruitless discussion about personal versioning scheme preference. My main point is: both schemes are widely used (even in the python world). As far as i can see both schemes can just co-exist in the python world giving us the benefits of both without hurting us. Short re-cap of both versioning schemes ======================================= The current version format specified in PEP 440 [2] follows this pseudo format: [N:]N(.N)*[{a|b|c|rc}N][.postN][.devN] So python module version numbers look like (taken from [2]): 1.0.dev456 1.0a1 1.0a2.dev456 1.0a12.dev456 1.0a12 1.0b1.dev456 1.0b2 1.0b2.post345.dev456 1.0b2.post345 1.0c1.dev456 1.0c1 1.0 1.0.post456.dev34 1.0.post456 1.1.dev1 Semantic Versioning follows this pseudo format: N.N.N(-((N)|([0-9A-Za-z-]+))(.((N)|([0-9A-Za-z-])+))*)? Some examples in order (taken from http://semver.org ): 1.0.0-alpha 1.0.0-alpha.1 1.0.0-alpha.beta 1.0.0-beta 1.0.0-beta.2 1.0.0-beta.11 1.0.0-rc.1 1.0.0 Key differences =============== Semantic Versioning supports - free-text pre-releases without version number such as '-alpha', '-pre' and the very widely used '-dev' (after a release the next commit increases the version number and appends the '-dev', which is only removed for the release). - always has MAJOR.MINOR.PATCH, (so 3 relevant) numbers as version number and offers guidelines which to change when. Semantic Versioning does not support - post-releases (a post release would be an increase of the PATCH number). - special handling for '-dev', '-alpha', '-beta', '-gamma' or 'rc'. Ideas to solve (cross scheme) comparisons ========================================= A version comparator could first try parsing both versions to be compared according to the current scheme and if that fails try parsing them both as SemanticVersions. Switching from one version naming scheme to another should be discouraged at least within the leading N parts at the change boundary staying the same. That way comparisons between the naming schemes could be settled by comparing the leading numerical parts of the version. Cheers, J?rn From guido at python.org Mon Apr 14 19:22:44 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2014 13:22:44 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: What does "official support" mean? It might be more productive if you had some code that you wanted to adopt, with an explanation of why it should go into the stdlib. On Mon, Apr 14, 2014 at 11:51 AM, J?rn Hees wrote: > Hi, > > what do you think about officially supporting Semantic Versioning? > ( http://semver.org ) > > Semantic Versioning has gained a lot of attention in other programming > languages. Even though not officially supported, many python libraries use > it > as well (just search the pypi listing for things like "-alpha" "-beta" > "-dev" > "-pre" to get a rough idea). There even is a class handling Semantic > Versioning > in pip already: pip.util.version.SemanticVersion . > > I'd speak in favor of officially adding support for semantic versioning to > the > python module versioning specs as a follow up to PEP 440 > ( http://legacy.python.org/dev/peps/pep-0440/ ). > > I want to avoid the fruitless discussion about personal versioning scheme > preference. > My main point is: both schemes are widely used (even in the python world). > As far as i can see both schemes can just co-exist in the python world > giving > us the benefits of both without hurting us. > > > > Short re-cap of both versioning schemes > ======================================= > > The current version format specified in PEP 440 [2] follows this pseudo > format: > [N:]N(.N)*[{a|b|c|rc}N][.postN][.devN] > > So python module version numbers look like (taken from [2]): > 1.0.dev456 > 1.0a1 > 1.0a2.dev456 > 1.0a12.dev456 > 1.0a12 > 1.0b1.dev456 > 1.0b2 > 1.0b2.post345.dev456 > 1.0b2.post345 > 1.0c1.dev456 > 1.0c1 > 1.0 > 1.0.post456.dev34 > 1.0.post456 > 1.1.dev1 > > > Semantic Versioning follows this pseudo format: > N.N.N(-((N)|([0-9A-Za-z-]+))(.((N)|([0-9A-Za-z-])+))*)? > > Some examples in order (taken from http://semver.org ): > 1.0.0-alpha > 1.0.0-alpha.1 > 1.0.0-alpha.beta > 1.0.0-beta > 1.0.0-beta.2 > 1.0.0-beta.11 > 1.0.0-rc.1 > 1.0.0 > > > > Key differences > =============== > > Semantic Versioning supports > - free-text pre-releases without version number such as '-alpha', '-pre' > and > the very widely used '-dev' (after a release the next commit increases the > version number and appends the '-dev', which is only removed for the > release). > - always has MAJOR.MINOR.PATCH, (so 3 relevant) numbers as version number > and > offers guidelines which to change when. > > Semantic Versioning does not support > - post-releases (a post release would be an increase of the PATCH number). > - special handling for '-dev', '-alpha', '-beta', '-gamma' or 'rc'. > > > > Ideas to solve (cross scheme) comparisons > ========================================= > > A version comparator could first try parsing both versions to be compared > according to the current scheme and if that fails try parsing them both as > SemanticVersions. > Switching from one version naming scheme to another should be discouraged > at > least within the leading N parts at the change boundary staying the same. > That > way comparisons between the naming schemes could be settled by comparing > the > leading numerical parts of the version. > > > > Cheers, > J?rn > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Apr 14 19:41:04 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 14 Apr 2014 13:41:04 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: So what i?d like to have happen (this part is here) is that PEP440 describes the syntax of versions in a way that enables as many possible versions for what is already on PyPI and that provides a definitive way for tools to parse, and interpret the version strings. Then what i?d like to have happen (this part hasn?t happened yet) is another PEP or document of some sort recommending the ?sane? subset of PEP440 and pushes people towards semantic like versioning within the constraints of PEP440. As far as just straight out adopting semver, it?s unlikely to happen because while the semantics match what a lot of people do, the syntax doesn?t. On Apr 14, 2014, at 1:22 PM, Guido van Rossum wrote: > What does "official support" mean? It might be more productive if you had some code that you wanted to adopt, with an explanation of why it should go into the stdlib. > > > On Mon, Apr 14, 2014 at 11:51 AM, J?rn Hees wrote: > Hi, > > what do you think about officially supporting Semantic Versioning? > ( http://semver.org ) > > Semantic Versioning has gained a lot of attention in other programming > languages. Even though not officially supported, many python libraries use it > as well (just search the pypi listing for things like "-alpha" "-beta" "-dev" > "-pre" to get a rough idea). There even is a class handling Semantic Versioning > in pip already: pip.util.version.SemanticVersion . > > I'd speak in favor of officially adding support for semantic versioning to the > python module versioning specs as a follow up to PEP 440 > ( http://legacy.python.org/dev/peps/pep-0440/ ). > > I want to avoid the fruitless discussion about personal versioning scheme > preference. > My main point is: both schemes are widely used (even in the python world). > As far as i can see both schemes can just co-exist in the python world giving > us the benefits of both without hurting us. > > > > Short re-cap of both versioning schemes > ======================================= > > The current version format specified in PEP 440 [2] follows this pseudo format: > [N:]N(.N)*[{a|b|c|rc}N][.postN][.devN] > > So python module version numbers look like (taken from [2]): > 1.0.dev456 > 1.0a1 > 1.0a2.dev456 > 1.0a12.dev456 > 1.0a12 > 1.0b1.dev456 > 1.0b2 > 1.0b2.post345.dev456 > 1.0b2.post345 > 1.0c1.dev456 > 1.0c1 > 1.0 > 1.0.post456.dev34 > 1.0.post456 > 1.1.dev1 > > > Semantic Versioning follows this pseudo format: > N.N.N(-((N)|([0-9A-Za-z-]+))(.((N)|([0-9A-Za-z-])+))*)? > > Some examples in order (taken from http://semver.org ): > 1.0.0-alpha > 1.0.0-alpha.1 > 1.0.0-alpha.beta > 1.0.0-beta > 1.0.0-beta.2 > 1.0.0-beta.11 > 1.0.0-rc.1 > 1.0.0 > > > > Key differences > =============== > > Semantic Versioning supports > - free-text pre-releases without version number such as '-alpha', '-pre' and > the very widely used '-dev' (after a release the next commit increases the > version number and appends the '-dev', which is only removed for the release). > - always has MAJOR.MINOR.PATCH, (so 3 relevant) numbers as version number and > offers guidelines which to change when. > > Semantic Versioning does not support > - post-releases (a post release would be an increase of the PATCH number). > - special handling for '-dev', '-alpha', '-beta', '-gamma' or 'rc'. > > > > Ideas to solve (cross scheme) comparisons > ========================================= > > A version comparator could first try parsing both versions to be compared > according to the current scheme and if that fails try parsing them both as > SemanticVersions. > Switching from one version naming scheme to another should be discouraged at > least within the leading N parts at the change boundary staying the same. That > way comparisons between the naming schemes could be settled by comparing the > leading numerical parts of the version. > > > > Cheers, > J?rn > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Mon Apr 14 19:42:01 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 14 Apr 2014 13:42:01 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: <82591C6C-ABBC-42CD-B2B1-E9BC5BF0C5F0@stufft.io> Oh also, this would probably be more appropriate for the distutils-sig mailing list. On Apr 14, 2014, at 1:41 PM, Donald Stufft wrote: > So what i?d like to have happen (this part is here) is that PEP440 describes > the syntax of versions in a way that enables as many possible versions for > what is already on PyPI and that provides a definitive way for tools to parse, > and interpret the version strings. > > Then what i?d like to have happen (this part hasn?t happened yet) is another > PEP or document of some sort recommending the ?sane? subset of PEP440 > and pushes people towards semantic like versioning within the constraints > of PEP440. > > As far as just straight out adopting semver, it?s unlikely to happen because > while the semantics match what a lot of people do, the syntax doesn?t. > > On Apr 14, 2014, at 1:22 PM, Guido van Rossum wrote: > >> What does "official support" mean? It might be more productive if you had some code that you wanted to adopt, with an explanation of why it should go into the stdlib. >> >> >> On Mon, Apr 14, 2014 at 11:51 AM, J?rn Hees wrote: >> Hi, >> >> what do you think about officially supporting Semantic Versioning? >> ( http://semver.org ) >> >> Semantic Versioning has gained a lot of attention in other programming >> languages. Even though not officially supported, many python libraries use it >> as well (just search the pypi listing for things like "-alpha" "-beta" "-dev" >> "-pre" to get a rough idea). There even is a class handling Semantic Versioning >> in pip already: pip.util.version.SemanticVersion . >> >> I'd speak in favor of officially adding support for semantic versioning to the >> python module versioning specs as a follow up to PEP 440 >> ( http://legacy.python.org/dev/peps/pep-0440/ ). >> >> I want to avoid the fruitless discussion about personal versioning scheme >> preference. >> My main point is: both schemes are widely used (even in the python world). >> As far as i can see both schemes can just co-exist in the python world giving >> us the benefits of both without hurting us. >> >> >> >> Short re-cap of both versioning schemes >> ======================================= >> >> The current version format specified in PEP 440 [2] follows this pseudo format: >> [N:]N(.N)*[{a|b|c|rc}N][.postN][.devN] >> >> So python module version numbers look like (taken from [2]): >> 1.0.dev456 >> 1.0a1 >> 1.0a2.dev456 >> 1.0a12.dev456 >> 1.0a12 >> 1.0b1.dev456 >> 1.0b2 >> 1.0b2.post345.dev456 >> 1.0b2.post345 >> 1.0c1.dev456 >> 1.0c1 >> 1.0 >> 1.0.post456.dev34 >> 1.0.post456 >> 1.1.dev1 >> >> >> Semantic Versioning follows this pseudo format: >> N.N.N(-((N)|([0-9A-Za-z-]+))(.((N)|([0-9A-Za-z-])+))*)? >> >> Some examples in order (taken from http://semver.org ): >> 1.0.0-alpha >> 1.0.0-alpha.1 >> 1.0.0-alpha.beta >> 1.0.0-beta >> 1.0.0-beta.2 >> 1.0.0-beta.11 >> 1.0.0-rc.1 >> 1.0.0 >> >> >> >> Key differences >> =============== >> >> Semantic Versioning supports >> - free-text pre-releases without version number such as '-alpha', '-pre' and >> the very widely used '-dev' (after a release the next commit increases the >> version number and appends the '-dev', which is only removed for the release). >> - always has MAJOR.MINOR.PATCH, (so 3 relevant) numbers as version number and >> offers guidelines which to change when. >> >> Semantic Versioning does not support >> - post-releases (a post release would be an increase of the PATCH number). >> - special handling for '-dev', '-alpha', '-beta', '-gamma' or 'rc'. >> >> >> >> Ideas to solve (cross scheme) comparisons >> ========================================= >> >> A version comparator could first try parsing both versions to be compared >> according to the current scheme and if that fails try parsing them both as >> SemanticVersions. >> Switching from one version naming scheme to another should be discouraged at >> least within the leading N parts at the change boundary staying the same. That >> way comparisons between the naming schemes could be settled by comparing the >> leading numerical parts of the version. >> >> >> >> Cheers, >> J?rn >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dev at joernhees.de Mon Apr 14 20:25:40 2014 From: dev at joernhees.de (=?windows-1252?Q?J=F6rn_Hees?=) Date: Mon, 14 Apr 2014 20:25:40 +0200 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: <827C6B09-881E-4C78-AD2E-5546929449BA@joernhees.de> On 14 Apr 2014, at 19:22, Guido van Rossum wrote: > What does "official support" mean? as i wrote: > I'd speak in favor of officially adding support for semantic versioning to the > python module versioning specs as a follow up to PEP 440 > ( http://legacy.python.org/dev/peps/pep-0440/ ). Currently if you write a library and search for how to do version numbering in python you?ll probably find PEP 440. I think it would be cool if either that PEP or a follow-up allowed (maybe even recommended) http://semver.org . So i?m talking about changing the specs, not stdlib code. Sorry that i didn?t make that clear. > It might be more productive if you had some code that you wanted to adopt, with an explanation of why it should go into the stdlib. Code actually is there already in pip.util.version.SemanticVersion . One could argue to move it to distutils, but i first wanted to evaluate if there?s support for allowing Semantic Versioning. j From dev at joernhees.de Mon Apr 14 20:28:36 2014 From: dev at joernhees.de (=?windows-1252?Q?J=F6rn_Hees?=) Date: Mon, 14 Apr 2014 20:28:36 +0200 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> On 14 Apr 2014, at 19:41, Donald Stufft wrote: > So what i?d like to have happen (this part is here) is that PEP440 describes > the syntax of versions in a way that enables as many possible versions for > what is already on PyPI and that provides a definitive way for tools to parse, > and interpret the version strings. Actually one of my points was that PEP440 currently describes versioning incompatible with http://semver.org and that at the same time many python libs on pypi use semver. j From guido at python.org Mon Apr 14 23:05:34 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2014 17:05:34 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> Message-ID: The funny thing is that the version parsing that's currently in distutils came out a similar discussion probably 15 or so years ago. There's really not much news in semver.org except that it has its own website. On Mon, Apr 14, 2014 at 2:28 PM, J?rn Hees wrote: > On 14 Apr 2014, at 19:41, Donald Stufft wrote: > > > So what i?d like to have happen (this part is here) is that PEP440 > describes > > the syntax of versions in a way that enables as many possible versions > for > > what is already on PyPI and that provides a definitive way for tools to > parse, > > and interpret the version strings. > > Actually one of my points was that PEP440 currently describes versioning > incompatible with http://semver.org and that at the same time many python > libs on pypi use semver. > > j > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Apr 14 23:35:19 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 14 Apr 2014 17:35:19 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> Message-ID: The PEP does currently have a semver section. In short, you can't really use the -suffix+anothersuffix versions, but you can have x.y.z versions where the major version number indicates a backwards incompatible change. On Mon, Apr 14, 2014 at 5:05 PM, Guido van Rossum wrote: > The funny thing is that the version parsing that's currently in distutils > came out a similar discussion probably 15 or so years ago. There's really > not much news in semver.org except that it has its own website. > > > On Mon, Apr 14, 2014 at 2:28 PM, J?rn Hees wrote: >> >> On 14 Apr 2014, at 19:41, Donald Stufft wrote: >> >> > So what i?d like to have happen (this part is here) is that PEP440 >> > describes >> > the syntax of versions in a way that enables as many possible versions >> > for >> > what is already on PyPI and that provides a definitive way for tools to >> > parse, >> > and interpret the version strings. >> >> Actually one of my points was that PEP440 currently describes versioning >> incompatible with http://semver.org and that at the same time many python >> libs on pypi use semver. >> >> j >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From leewangzhong+python at gmail.com Mon Apr 14 23:44:17 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 14 Apr 2014 17:44:17 -0400 Subject: [Python-ideas] A namedtuple literal. Message-ID: (Attempting to reply to https://mail.python.org/pipermail/python-ideas/2014-April/027443.html) > In JS, if you want to define a simple object, you can write it in two ways: > {'spam': 'ham'} > or > {spam: 'ham'} > But when you have a key name defined in a variable, you'll need to do > key = 'spam' > o = {} > o[key] = 'ham' > Where in Python, you'd simply write {key: 'ham'}. > So for Python, I think that having unquoted keys in literals is a bad idea. I agree with this (even if you no longer do). There's no precedence (that I can think of) for NOT evaluating things before a colon, and nothing like using attr names (rather than objects that hold attribute names) with a colon. As long as we're just fooling around with ideas, here's an alternative proposal for namedtuple literal syntax[0]. I don't mean this as an actual suggestion, but I think it is better than unquoted keys before colons. ('color':c, 'position':c) #quoted, or with variables (color=c, position=c) #unquoted, with literals Example: p = (x=1, y=5) #type is something new, like `ntuple`, or just `namedtuple`. This is similar to keyword arguments, which don't have to quote their first parts. This can be generalized to dicts, to keep consistency. d = {'hello':1, 'world':2} #original d = {hello=1, world=2} #new and the earlier-proposed "ordered dict literal"[1] od = ['hello':1, 'world':2] #new od = [hello=1, world=2] #new new Or alternatively, with `=` just appearing in ntuples, this can be used: od = OrderedDict((hello=1, world=2)) which is interesting and also somehow horrifying. Some arguments against this: - Introducing `=` in yet another context. Ugly. - Tuples are ordered, and **kwargs are unordered[2], so there's a conceptual discontinuity there. - Accidentally writing `:` when you mean `=` can lead to silent bugs. - Extending the syntax to dictionaries possibly violates TOOWTDI, and not doing so feels inconsistent to me. I think about `(var_holding_spam: 1, literally_eggs=2)` and I'm like, "This is obviously two ways of doing it." - People might expect `p = (x=1) to be a 1-ntuple. I don't want it to be, and I don't like that it wouldn't be. I like the taste of the syntax (and the generalizations, and the OrderedDict literal), but I see how it isn't necessarily necessary (or welcome). [0] Found a mention of this here: https://mail.python.org/pipermail/python-ideas/2010-October/008489.html [1] Guido hated that: https://mail.python.org/pipermail/python-ideas/2009-June/004924.html [2] Making it ordered is obstructed by concerns about performance ( https://mail.python.org/pipermail/python-ideas/2013-February/019699.html). Currently being discussed here: https://mail.python.org/pipermail/python-ideas/2014-April/027454.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at erisian.com.au Tue Apr 15 01:25:28 2014 From: aj at erisian.com.au (Anthony Towns) Date: Tue, 15 Apr 2014 09:25:28 +1000 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: On 15 April 2014 07:44, Franklin? Lee wrote: > As long as we're just fooling around with ideas, here's an alternative > proposal for namedtuple literal syntax[0]. Isn't a namedtuple literal a bad idea? Isn't the point of a namedtuple (as distinct from a dict or OrderedDict), that each object only stores the data and the names are stored in the type, making having lots of namedtuples more memory efficient? ("Named tuple instances do not have per-instance dictionaries, so they are lightweight and require no more memory than regular tuples." -- https://docs.python.org/2/library/collections.html#collections.namedtuple) If you already have a specific namedtuple type instantiated, literals are easy: Point = namedtuple('Point', ['x', 'y']) Point(1,2) # positional Point(y=2, x=1) # named > - People might expect `p = (x=1) to be a 1-ntuple. I don't want it to be, > and I don't like that it wouldn't be. Adding a trailing comma like with normal 1-tuples to get "p = (x=1,)" would seem like the obvious approach? a=(b=1) is already invalid syntax though. Cheers, aj -- Anthony Towns From dev at joernhees.de Tue Apr 15 02:30:56 2014 From: dev at joernhees.de (=?windows-1252?Q?J=F6rn_Hees?=) Date: Tue, 15 Apr 2014 02:30:56 +0200 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> Message-ID: <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> On 14 Apr 2014, at 23:35, Daniel Holth wrote: > The PEP does currently have a semver section. In short, you can't > really use the -suffix+anothersuffix versions, but you can have x.y.z > versions where the major version number indicates a backwards > incompatible change. yes, there is such a section, but my point is exactly that where the current spec points out that ?Semantic versioning is a popular version identification scheme? and then explains where it?s incompatible, we should just allow it as version specs. Currently there are about 1 % of all version numbers on pypi which show exactly the incompatibility described in http://legacy.python.org/dev/peps/pep-0440/#semantic-versioning : I find 432 matches for the regexp "^\d+.\d+.\d+-? and reviewing them i?m quite convinced that most of them follow semver. Overall we have 45760 version strings in pypi: [1] 28989 out of the current versions are just 3 numbers (?^\d+.\d+.\d+$?) and could qualify as semver as well. To me the fact that somewhat between 1 and 64 % of the packages in pypi follow semver is enough to ask why we don?t fully support it and maybe even encourage it. And yes there is "not much news in semver.org except that it has its own website?, but it?s a pretty short, convincing and easy to understand website. I guess that?s why so many people refer to it. j [1]: https://gist.githubusercontent.com/joernhees/10688223/raw/2998db303f3221cf594922bed5a36bcaefc867d3/pypi_versions.txt From donald at stufft.io Tue Apr 15 02:46:06 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 14 Apr 2014 20:46:06 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> Message-ID: PEP440 and semver are incompatible, however at the point we?re parsing versions there isn?t a good way for a particular project to say ?hey we?re using semver? or ?hey we?re using PEP440?. So we pretty much need a singular version syntax, and PEP440 covers almost everything already on PyPI. On Apr 14, 2014, at 8:30 PM, J?rn Hees wrote: > On 14 Apr 2014, at 23:35, Daniel Holth wrote: > >> The PEP does currently have a semver section. In short, you can't >> really use the -suffix+anothersuffix versions, but you can have x.y.z >> versions where the major version number indicates a backwards >> incompatible change. > > yes, there is such a section, but my point is exactly that where the current spec points out that ?Semantic versioning is a popular version identification scheme? and then explains where it?s incompatible, we should just allow it as version specs. > > Currently there are about 1 % of all version numbers on pypi which show exactly the incompatibility described in http://legacy.python.org/dev/peps/pep-0440/#semantic-versioning : > I find 432 matches for the regexp "^\d+.\d+.\d+-? and reviewing them i?m quite convinced that most of them follow semver. > Overall we have 45760 version strings in pypi: [1] > 28989 out of the current versions are just 3 numbers (?^\d+.\d+.\d+$?) and could qualify as semver as well. > > To me the fact that somewhat between 1 and 64 % of the packages in pypi follow semver is enough to ask why we don?t fully support it and maybe even encourage it. > > And yes there is "not much news in semver.org except that it has its own website?, but it?s a pretty short, convincing and easy to understand website. I guess that?s why so many people refer to it. > > j > > [1]: https://gist.githubusercontent.com/joernhees/10688223/raw/2998db303f3221cf594922bed5a36bcaefc867d3/pypi_versions.txt > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From leewangzhong+python at gmail.com Tue Apr 15 03:18:04 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 14 Apr 2014 21:18:04 -0400 Subject: [Python-ideas] A namedtuple literal. In-Reply-To: References: Message-ID: I don't see much of a necessary memory issue with namedtuple literals. Upon creation of an anonymous namedtuple, you can map the tuple of names to a single type object. Sample pseudo-code (*smirk*) implementation for what the interpreter could do, given the key-value pairs: def ntuple(pairs: OrderedDict): names = tuple(pairs.keys()) try: tuple_factory = existing_ntuples[names] except ValueError: tuple_factory = namedtuple(None, names) # `None` not actually legal for namedtuple's name param existing_ntuples[names] = tuple_factory return tuple_factory(*pairs.values()) A different (ideological?) problem: namedtuples look like they're basically made to be both tuples AND simple lightweight immutable *classes*. The namedtuple literal implemented this way would be effectively allowing reusable anonymous classes. I don't see too much of a problem with this, though, since we should usually not care if they're ducks or really monkeys. Here are some sample "abuses", though: a = (x=4, y=5) b = type(a)(6, 7) c = (x='snark', y='sarcasm') type(c) == type(b) #True Actually, now that I write it out, I like that the type of an anonymous named tuple depends on the property names, and that two anonymous ntuples with different keylists are different. On Mon, Apr 14, 2014 at 7:25 PM, Anthony Towns wrote: > On 15 April 2014 07:44, Franklin? Lee > wrote: > > As long as we're just fooling around with ideas, here's an alternative > > proposal for namedtuple literal syntax[0]. > > Isn't a namedtuple literal a bad idea? Isn't the point of a namedtuple > (as distinct from a dict or OrderedDict), that each object only stores > the data and the names are stored in the type, making having lots of > namedtuples more memory efficient? ("Named tuple instances do not have > per-instance dictionaries, so they are lightweight and require no more > memory than regular tuples." -- > https://docs.python.org/2/library/collections.html#collections.namedtuple) > > If you already have a specific namedtuple type instantiated, literals are > easy: > > Point = namedtuple('Point', ['x', 'y']) > Point(1,2) # positional > Point(y=2, x=1) # named > > > - People might expect `p = (x=1) to be a 1-ntuple. I don't want it to be, > > and I don't like that it wouldn't be. > > Adding a trailing comma like with normal 1-tuples to get "p = (x=1,)" > would seem like the obvious approach? a=(b=1) is already invalid > syntax though. > > Cheers, > aj > > -- > Anthony Towns > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Apr 15 03:50:00 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 15 Apr 2014 10:50:00 +0900 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> Message-ID: <87vbubfjjr.fsf@uwakimon.sk.tsukuba.ac.jp> J?rn Hees writes: > Currently there are about 1 % of all version numbers on pypi which > show exactly the incompatibility described in > http://legacy.python.org/dev/peps/pep-0440/#semantic-versioning : > To me the fact that somewhat between 1 and 64 % of the packages in > pypi follow semver is enough to ask why we don?t fully support it So, as I understand what you're saying, PEP 440 *does* support a large subset of the semantic versioning actually used on PyPI (63/64 = 98.5%). I find that simple math a pretty compelling reason not to make the PEP 440 spec more complicated. > and maybe even encourage it. The fact that semantic versioning would be a good, even great, standard if it were the *only* standard is not compelling. "The One Standard to rule them all" is a mirage: "The great thing about 'standards' is that there are so many to choose from!" Worse, there are so many that *have already been chosen*. From leewangzhong+python at gmail.com Tue Apr 15 04:20:48 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 14 Apr 2014 22:20:48 -0400 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) Message-ID: Would string interning help? As in, all keywords in a **pack have to be strings. All parameter names can be interned (if they're not already). The **pack can be a special dict optimized for strings (more specifically: identifiers), with an order-preserving overhead. (And then **unpacking would be smart about the type of thing being unpacked.) I understand that there would need to be a C proof-of-concept that this could act like a dict on the Python level, and maybe even beat a dict for **kwargs performance. I'm not suggesting that it's viable, but I haven't seen a mention of taking advantage of the restrictions on the keys, and while I personally got stuck trying to flesh out an algorithm and structure for it (unnecessary(?) overhead on extracting some keys and passing on the rest while preserving order), maybe someone more clever can figure out how to work it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Tue Apr 15 04:56:12 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 14 Apr 2014 22:56:12 -0400 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: Message-ID: Sigh. I couldn't figure out how to get GMail to fake its way into a thread for emails I never got. How embarrassing. Was trying to reply in this thread: https://mail.python.org/pipermail/python-ideas/2014-April/027454.html Anyway, I understand that Raymond Hettinger is implementing a better dict that would inadvertently be ordered ( https://mail.python.org/pipermail/python-dev/2012-December/123028.html). I am just wondering about whether the hashing algorithm or internal shape of the automatic dictionary could be improved given what the interpreter can guarantee about the keyspace. On Mon, Apr 14, 2014 at 10:20 PM, Franklin? Lee < leewangzhong+python at gmail.com> wrote: > Would string interning help? > > As in, all keywords in a **pack have to be strings. All parameter names > can be interned (if they're not already). The **pack can be a special dict > optimized for strings (more specifically: identifiers), with an > order-preserving overhead. (And then **unpacking would be smart about the > type of thing being unpacked.) > > I understand that there would need to be a C proof-of-concept that this > could act like a dict on the Python level, and maybe even beat a dict for > **kwargs performance. I'm not suggesting that it's viable, but I haven't > seen a mention of taking advantage of the restrictions on the keys, and > while I personally got stuck trying to flesh out an algorithm and structure > for it (unnecessary(?) overhead on extracting some keys and passing on the > rest while preserving order), maybe someone more clever can figure out how > to work it. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Apr 15 05:11:53 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 14 Apr 2014 20:11:53 -0700 (PDT) Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: Message-ID: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> From: Franklin? Lee Sent: Monday, April 14, 2014 7:20 PM >Would string interning help? > >As in, all keywords in a **pack have to be strings.?All parameter names can be interned (if they're not already). The **pack can be a special dict optimized for strings (more specifically: identifiers), with an order-preserving overhead. (And then **unpacking would be smart about the type of thing being unpacked.) The dict unpacked into keyword arguments is not the same as the dict passed to the keyword-dict parameter. Some trivial examples: ? ? def spam(a): ? ? ? ? pass ? ? spam(**{'a': 1}) ? ? def eggs(**kw): ? ? ? ? pass ? ? eggs(a=1) ? ? def beans(**kw): ? ? ? ? return kw ? ? d = {'a': 1} ? ? assert beans(**d) is not d Also, I think you've got packing and unpacking backward. There's no way **unpack can be smart about the type of thing being unpacked, because it explicitly allows for any kind of mapping, supplied by the user, and unpacks its keys as if they were keyword arguments. Anyway, even if things did work this way, one of Guido's objections is that he writes lots of code that keeps the kwargs dict around and adds to it later. (Presumably he has a method that takes **kw and stores it in an instance attribute, although he didn't give examples.) Using something which is not a dict, but only fakes it just well enough for passing keyword arguments, surely isn't going to satisfy him. And other people do things to their kwargs dicts that you'd have to support. For example, there's plenty of Python 2.x code that fakes keyword-only arguments by calling kwargs.pop; that code still runs in Python 3.4, and I don't think you want to break it in 3.5. So you're effectively going to have to build a full MutableMapping. Finally: >I understand that there would need to be a C proof-of-concept that? >this could act like a dict on the Python level, and maybe even beat a dict for **kwargs performance. You really think so? First, I don't know that hashing the strings (which can be cached!) is the slow part of this. But, if it is, hashing the strings to put them into an interned string table, and then hashing those interned pointers to put into the mini-dict, isn't going to be faster. It could be _smaller_, but I don't think anyone cares about the space used for a kwargs dict during the function calling machinery.? Guido _does_ care about the space used for a kwargs dict that he chooses to save inside the function and later add a bunch of stuff to. But if you're worried about that case, providing something that's kind of like a dict but not is just wrong. This code has to work: ? ? class Spam: ? ? ? ? def __init__(self, **kwargs): ? ? ? ? ? ? self.config = kwargs ? ? ? ? def config(self, key, value): ? ? ? ? ? ? self.config[key] = value ? ? spam = Spam(count=10) ? ? spam.config(42, 'forty-two') So you're talking about implementing the entire MutableMapping interface, at which point I don't think you have a mini-dict at all. From leewangzhong+python at gmail.com Tue Apr 15 06:12:11 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 15 Apr 2014 00:12:11 -0400 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: ?> Also, I think you've got packing and unpacking backward. There's no way **unpack can be smart about the type of thing being unpacked, because it explicitly allows for any kind of mapping, supplied by the user, and unpacks its keys as if they were keyword arguments. No, I mean that the interpreter can inspect the type of the kwargs during unpacking. Primarily, it can pass along the same type (I know it shouldn't be the same *object*). It's also possible that, in the future, it can pass a *view* of the kwargs dictionary *if* the callee would receive the exact same keys (or maybe even more generally), and then only do a deeper copy of the item pointers and order array if either the view or the original is changed. > Anyway, even if things did work this way, one of Guido's objections is that he writes lots of code that keeps the kwargs dict around and adds to it later. (Presumably he has a method that takes **kw and stores it in an instance attribute, although he didn't give examples.) Using something which is not a dict, but only fakes it just well enough for passing keyword arguments, surely isn't going to satisfy him. Well, what exactly do you mean "not a dict"? It won't be a literal dict type, no, but it might be a subclass. It would be better if the InternedDict could be as fast as possible (for the usual case), but I don't think that it would absolutely cripple the idea to make it Mutable ... A-and on the flip side, I put forth that maybe Guido shouldn't be doing that kind of thing in the first place (*gasp*)! If he wanted a copy, he should've explicitly copied it with dict(), instead of relying on behavior that I've never even seen a doc reference for! (Not sure if I actually believe that **kwargs should be some subclass of FrozenDict, but maybe someone else does, and they can convince us.) > For example, there's plenty of Python 2.x code that fakes keyword-only arguments by calling kwargs.pop; that code still runs in Python 3.4, and I don't think you want to break it in 3.5. So you're effectively going to have to build a full MutableMapping. So... you're saying there's a chance we can break it in 3.6, right? No reason not to look to the future if backwards compatibility is what's keeping it from happening now. > You really think so? Yes, I believe that it is not impossible that someone knows how to make pointer lookups faster than general hash lookups (a function call) of general dicts. This may be because I haven't looked at the dict implementation, or because the dict implementation could be better (see https://mail.python.org/pipermail/python-dev/2012-December/123028.html), but I'm putting forth the idea because maybe it's not impossible, and someone can make it possible. > But, if it is, hashing the strings to put them into an interned string table, and then hashing those interned pointers to put into the mini-dict, isn't going to be faster. Why not? If an InternedDict is passing args to a newly-created IDict, the hashing could be a single instruction: modulo by 63 (or something). (And then maybe multiple instructions to crawl through a linked list.) And knowing you don't HAVE to call a str.hash is surely better than calling even a cached str.hash, no? Point is, again, I don't know that it won't be faster. You seem to know that it won't be faster, but I don't see you saying that there CAN'T be such cleverness when kwargs are nice and forwarded nicely, even with deletions or additions allowed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Apr 15 06:41:34 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 15 Apr 2014 00:41:34 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: On 4/14/2014 11:51 AM, J?rn Hees wrote: > what do you think about officially supporting Semantic Versioning? > ( http://semver.org ) > Key differences > =============== There are two types of differences. First are the syntax differences, which you discuss and illustrate. The top-level x.y.z syntax of semver is compatible with PEP440. The incompatibilites come in the postfix tags, and in particular, the use of '.' versus '-' following 'z'. Second are the semantic differences, which you did not discuss. PEP 440 does not prescribe a meaning for major, minor, micro(patch) numbers. Semver does. As I read semver.org, semver syntax without semver semantics is not Semantic Versioning. So officially supporting Semantic Versioning should mean adopting the semantics. Indeed, a project could use semver semantics for x.y.z while using PEP 440 syntax for postfix tags. For Python, the semantics would require that each Python version get a new major number, and that the minor number might always be 0. I don't see that as useful. Semantic semver would only be practical on a module by module basis (or in some cases, packages). As I said in response to this same post on python-list, one problem with mechanical semantic interpretation of version numbers, the supposed justification for the system, is that such interpretation does not work as soon as one accidentally makes a release that violates the semantic rules. -- Terry Jan Reedy From rosuav at gmail.com Tue Apr 15 10:09:49 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 15 Apr 2014 18:09:49 +1000 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: On Tue, Apr 15, 2014 at 2:12 PM, Franklin? Lee wrote: > ... A-and on the flip side, I put forth that maybe Guido shouldn't be doing > that kind of thing in the first place (*gasp*)! If he wanted a copy, he > should've explicitly copied it with dict(), instead of relying on behavior > that I've never even seen a doc reference for! (Not sure if I actually > believe that **kwargs should be some subclass of FrozenDict, but maybe > someone else does, and they can convince us.) A function's collected kwargs will always be a new dictionary. That's a language guarantee. There are a number of common idioms that make use of this, such as subclassing and consuming only certain args: def __init__(self, *, **kwargs): self.foo = kwargs.pop("foo") super().__init__(**kwargs) ChrisA From anthony at xtfx.me Tue Apr 15 12:11:18 2014 From: anthony at xtfx.me (C Anthony Risinger) Date: Tue, 15 Apr 2014 05:11:18 -0500 Subject: [Python-ideas] Literals call local constructors()? Message-ID: On Apr 14, 2014 4:44 PM, "Franklin? Lee" wrote: [...] > This is similar to keyword arguments, which don't have to quote their first parts. This can be generalized to dicts, to keep consistency. > d = {'hello':1, 'world':2} #original > d = {hello=1, world=2} #new This not working is unexpected, if considering the expectation literals: {..} and [...] ...translate to: dict(...) and list(...) [int(..) and str(..)] Why isn't/can't this be true? I discovered this awhile back while attempting to change the default constructor with: dict = OtherDict ...equiv: module.dict = OtherDict ...and: __builtins__['dict'] = OtherDict ...but none work :-( -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Apr 15 12:40:13 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 15 Apr 2014 20:40:13 +1000 Subject: [Python-ideas] Literals call local constructors()? In-Reply-To: References: Message-ID: On Tue, Apr 15, 2014 at 8:11 PM, C Anthony Risinger wrote: > This not working is unexpected, if considering the expectation literals: > > {..} and [...] > > ...translate to: > > dict(...) and list(...) [int(..) and str(..)] > > Why isn't/can't this be true? They don't. They translate into dict-creation bytecodes. In CPython: Python 3.5.0a0 (default:6a0def54c63d, Mar 26 2014, 01:11:09) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import dis >>> def make_dict1(): ... return {'hello':1, 'world':2} ... >>> def make_dict2(): ... return dict(hello=1, world=2) ... >>> dis.dis(make_dict1) 2 0 BUILD_MAP 2 3 LOAD_CONST 1 (1) 6 LOAD_CONST 2 ('hello') 9 STORE_MAP 10 LOAD_CONST 3 (2) 13 LOAD_CONST 4 ('world') 16 STORE_MAP 17 RETURN_VALUE >>> dis.dis(make_dict2) 2 0 LOAD_GLOBAL 0 (dict) 3 LOAD_CONST 1 ('hello') 6 LOAD_CONST 2 (1) 9 LOAD_CONST 3 ('world') 12 LOAD_CONST 4 (2) 15 CALL_FUNCTION 512 (0 positional, 2 keyword pair) 18 RETURN_VALUE If you want the name dict to be looked up, look up the name dict. Otherwise, code like this would be very VERY confusing: >>> def make_dict3(): ... dict = {'hello':1, 'world':2} ... return dict ... >>> dis.dis(make_dict3) 2 0 BUILD_MAP 2 3 LOAD_CONST 1 (1) 6 LOAD_CONST 2 ('hello') 9 STORE_MAP 10 LOAD_CONST 3 (2) 13 LOAD_CONST 4 ('world') 16 STORE_MAP 17 STORE_FAST 0 (dict) 3 20 LOAD_FAST 0 (dict) 23 RETURN_VALUE If that had to actually call dict(), it would raise UnboundLocalError for something that doesn't seem to reference locals before assigning to them. As it is, it's exactly the same as make_dict1 except that it does a store/load unnecessarily. ChrisA From dev at joernhees.de Tue Apr 15 14:22:49 2014 From: dev at joernhees.de (=?windows-1252?Q?J=F6rn_Hees?=) Date: Tue, 15 Apr 2014 14:22:49 +0200 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: <87vbubfjjr.fsf@uwakimon.sk.tsukuba.ac.jp> References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> <87vbubfjjr.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 15 Apr 2014, at 03:50, Stephen J. Turnbull wrote: > So, as I understand what you're saying, PEP 440 *does* support a large > subset of the semantic versioning actually used on PyPI (63/64 = > 98.5%). I find that simple math a pretty compelling reason not to > make the PEP 440 spec more complicated. You can think of it that way. I?m more worried because i see it the other way around: I suspect that up to 64 % of the projects actually follow semver (1 % for sure) and only by chance follow PEP 440 ;). Encouraging semver in PEP 440 would in my opinion have made versioning of python modules more meaningful and compatible with a widely used ?standard? across languages. Anyhow, i can see that there?s not much support for this, but at least i tried bringing it up. Thanks for the feedback. Cheers, J?rn From ericsnowcurrently at gmail.com Tue Apr 15 15:01:01 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 15 Apr 2014 07:01:01 -0600 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: On Mon, Apr 14, 2014 at 10:12 PM, Franklin? Lee wrote: > Point is, again, I don't know that it won't be faster. You seem to know that > it won't be faster, but I don't see you saying that there CAN'T be such > cleverness when kwargs are nice and forwarded nicely, even with deletions or > additions allowed. The point is moot. The object into which uncollected kwargs is bound must be a complete mapping instance that supports any type of key. We certainly *initialize* it with string keys, but once it is bound in the function, code is free to do anything that works with a normal dict (think Liskov). Limiting capability is a non-starter. That said, thanks for posting the idea. It wasn't all that unfeasible (unlike other ideas that land here ), even if it ultimately wasn't acceptable. -eric From ncoghlan at gmail.com Tue Apr 15 15:15:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Apr 2014 09:15:05 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: On 15 Apr 2014 00:42, "Terry Reedy" wrote: > > On 4/14/2014 11:51 AM, J?rn Hees wrote: > >> what do you think about officially supporting Semantic Versioning? >> ( http://semver.org ) > > >> Key differences >> =============== > > > There are two types of differences. First are the syntax differences, which you discuss and illustrate. The top-level x.y.z syntax of semver is compatible with PEP440. The incompatibilites come in the postfix tags, and in particular, the use of '.' versus '-' following 'z'. > > Second are the semantic differences, which you did not discuss. PEP 440 does not prescribe a meaning for major, minor, micro(patch) numbers. Semver does. As I read semver.org, semver syntax without semver semantics is not Semantic Versioning. So officially supporting Semantic Versioning should mean adopting the semantics. Indeed, a project could use semver semantics for x.y.z while using PEP 440 syntax for postfix tags. > > For Python, the semantics would require that each Python version get a new major number, and that the minor number might always be 0. I don't see that as useful. Semantic semver would only be practical on a module by module basis (or in some cases, packages). PEP 440 uses semantic versioning semantics by default unless a project appears to be using date based versions, or the consumer specifies a different comparison operator. The only real discrepancy is the postfix syntax, and that's due to the unordered nature of semver build labels. Cheers, Nick. > > As I said in response to this same post on python-list, one problem with mechanical semantic interpretation of version numbers, the supposed justification for the system, is that such interpretation does not work as soon as one accidentally makes a release that violates the semantic rules. > > -- > Terry Jan Reedy > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Apr 15 15:10:25 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Apr 2014 09:10:25 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> <87vbubfjjr.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: A) wrong list (discussions-to in the PEP is set to distutils-sig for a reason - most of the packaging specialists are there, not here) B) No. PEP 440 deliberately requires total ordering of published versions to make downloads deterministic, and semver build labels don't support that. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Apr 15 17:37:38 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 15 Apr 2014 11:37:38 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: <263EB89C-E6BD-4907-BEF8-E7857F7E9290@joernhees.de> <0454973E-FB6E-4186-82BD-22388F5686C1@joernhees.de> <87vbubfjjr.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: PEP 440 is descended from PEP 386 (2009). When PEP 386 was new a lot of projects did change the way they do version numbers because of it. Much of the change consisted of dropping silly random suffixes from the version identifiers. Now it has been repeatedly bikeshedded to death multiple times, then bikeshedded again for good measure. I have come to the conclusion that the version number doesn't matter that much as long as it can be sorted. On Tue, Apr 15, 2014 at 9:10 AM, Nick Coghlan wrote: > A) wrong list (discussions-to in the PEP is set to distutils-sig for a > reason - most of the packaging specialists are there, not here) > > B) No. PEP 440 deliberately requires total ordering of published versions to > make downloads deterministic, and semver build labels don't support that. > > Regards, > Nick. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From leewangzhong+python at gmail.com Tue Apr 15 19:24:54 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 15 Apr 2014 13:24:54 -0400 Subject: [Python-ideas] Literals call local constructors()? In-Reply-To: References: Message-ID: I think C Anthony is saying that he tried to make it happen, and it didn't, but he figured something else out, which is what is relevant to him here. I am mostly sure he isn't arguing for the ability to hack dict literals On Tue, Apr 15, 2014 at 6:40 AM, Chris Angelico wrote: > On Tue, Apr 15, 2014 at 8:11 PM, C Anthony Risinger > wrote: > > This not working is unexpected, if considering the expectation literals: > > > > {..} and [...] > > > > ...translate to: > > > > dict(...) and list(...) [int(..) and str(..)] > > > > Why isn't/can't this be true? > > They don't. They translate into dict-creation bytecodes. In CPython: > > Python 3.5.0a0 (default:6a0def54c63d, Mar 26 2014, 01:11:09) > [GCC 4.7.2] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import dis > >>> def make_dict1(): > ... return {'hello':1, 'world':2} > ... > >>> def make_dict2(): > ... return dict(hello=1, world=2) > ... > >>> dis.dis(make_dict1) > 2 0 BUILD_MAP 2 > 3 LOAD_CONST 1 (1) > 6 LOAD_CONST 2 ('hello') > 9 STORE_MAP > 10 LOAD_CONST 3 (2) > 13 LOAD_CONST 4 ('world') > 16 STORE_MAP > 17 RETURN_VALUE > >>> dis.dis(make_dict2) > 2 0 LOAD_GLOBAL 0 (dict) > 3 LOAD_CONST 1 ('hello') > 6 LOAD_CONST 2 (1) > 9 LOAD_CONST 3 ('world') > 12 LOAD_CONST 4 (2) > 15 CALL_FUNCTION 512 (0 positional, 2 keyword pair) > 18 RETURN_VALUE > > If you want the name dict to be looked up, look up the name dict. > Otherwise, code like this would be very VERY confusing: > > >>> def make_dict3(): > ... dict = {'hello':1, 'world':2} > ... return dict > ... > >>> dis.dis(make_dict3) > 2 0 BUILD_MAP 2 > 3 LOAD_CONST 1 (1) > 6 LOAD_CONST 2 ('hello') > 9 STORE_MAP > 10 LOAD_CONST 3 (2) > 13 LOAD_CONST 4 ('world') > 16 STORE_MAP > 17 STORE_FAST 0 (dict) > > 3 20 LOAD_FAST 0 (dict) > 23 RETURN_VALUE > > If that had to actually call dict(), it would raise UnboundLocalError > for something that doesn't seem to reference locals before assigning > to them. As it is, it's exactly the same as make_dict1 except that it > does a store/load unnecessarily. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Tue Apr 15 20:12:34 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 15 Apr 2014 14:12:34 -0400 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: Well, a little effort on the last ditch. Like I said before, I don't see the need for modification to mean that this can't be made to work. I still think it would be possible to make the InternDict start off with the knowledge of the keys, and then upon attempt to add, it can either: - use a second internal array to store non-str keys (or just non-interned keys? or just all new keys?), and doing a check to see, "Am I trying to access a str key?". The hash would have an index into one of the two arrays. If it's storing new str keys in this new array (as opposed to only non-str keys), the interpreter would flatten again upon **blowup. (Example indexing for the second array: -1 is the first NEW element, then -2... If you ask for an element, and it's a str, then intern it and hash its interned ID and look it up on the hash table. If it's negative, it's in the second array. If you ask for the ordered keys, it will trek through the original array forwards, then the newarray backwards. Also, the standard case doesn't need to worry about lookup overhead, because it only looks for a second array if the hash table reports a negative index. It does need to check the second-array pointer if it's getting the elements in order.) - internally convert to a regular dict. This is undesirable, because it would lose order, which would be surprising to users. and upon delete, it would have to deal with deleting two entries (hash table and array). Fortunately, it wouldn't need to shift everything down, probably, and the unpacking call can just get the elements in sorted order. Both of these, alas, would cause additional overhead on add/delete, and the first causes extra bookkeeping even without that use (a pointer to a second internal array, even if that pointer is null, and occasional checking of that pointer when working with the whole table; and always intern strings going in). And if the kwargs are an OrderedDict, and not just acting like an ordered dict until you change it, it would have to remember the order of those new members, even if they're not going to be repacked into kwargs, and even if the user doesn't care about the order. But I don't know enough to say that that's a killer. As in, maybe it improves the usual case (calls) enough that the harm to the modifying case is cancelled out for big programs. Maybe non-modifying-case speedup + modifying-case slowdown < 0, and maybe if it looks enough like a dict, no user ever has to know. That's about five maybes, for those not keeping track. On Tue, Apr 15, 2014 at 9:01 AM, Eric Snow wrote: > On Mon, Apr 14, 2014 at 10:12 PM, Franklin? Lee > wrote: > > Point is, again, I don't know that it won't be faster. You seem to know > that > > it won't be faster, but I don't see you saying that there CAN'T be such > > cleverness when kwargs are nice and forwarded nicely, even with > deletions or > > additions allowed. > > The point is moot. The object into which uncollected kwargs is bound > must be a complete mapping instance that supports any type of key. We > certainly *initialize* it with string keys, but once it is bound in > the function, code is free to do anything that works with a normal > dict (think Liskov). Limiting capability is a non-starter. > > That said, thanks for posting the idea. It wasn't all that unfeasible > (unlike other ideas that land here ), even if it ultimately > wasn't acceptable. > > -eric > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Apr 15 20:56:47 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 15 Apr 2014 11:56:47 -0700 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: <48E80F4E-8EC0-4A69-950B-32990797EC3C@yahoo.com> On Apr 14, 2014, at 21:12, "Franklin? Lee" wrote: > ?> Also, I think you've got packing and unpacking backward. There's no way **unpack can be smart about the type of thing being unpacked, because it explicitly allows for any kind of mapping, supplied by the user, and unpacks its keys as if they were keyword arguments. > > No, I mean that the interpreter can inspect the type of the kwargs during unpacking. Primarily, it can pass along the same type (I know it shouldn't be the same *object*). So if the object is some hypothetical FrozenDict type like you mentioned below, it should create an instance of that type and then copy the values into it? Or what if someone unpacks a dbm? For that matter, I've got a custom mapping object that maps a handful of predefined keys to true or false by storing a bitmap; I'm not sure whether I've unpacked this into **kwargs, but it makes sense, it works today, and would work with the OrderedDict proposal, but obviously wouldn't work if you tried to create the same type for kwargs. More importantly, I think you're still missing the point that keyword unpacking and extra-keyword parameters are not directly connected. A **kwargs parameter is not generally used to accept a **mapping argument pack from the caller; if you want that, you just pass the dict as a normal argument. A **kwargs parameter is for collecting extra keyword arguments, and a **mapping is for supplying keyword arguments from an existing mapping. In the special case that both exist, there are no keyword arguments that don't match normal parameters, and there are no keys in the unpacked dict that do match normal parameter. In any other case, they have different contents. Look at the examples I gave before. For example, how is this going to preserve keyword order in cases where nobody unpacks a mapping along with the keywords? There is no object to create a new object of the same type. Or, putting both problems together: def spam(**kw): pass d = HypotheticalFrozenDict(b=1) spam(a=0, **d) What is kw going to receive? Also, even if this worked, you're putting the burden of ordering the keywords on the caller, rather than on the function that actually cares about the keyword order. That's the wrong place to put it. You've solved the case of forwarding delegate methods, partial, etc., but not the everyday case. > It's also possible that, in the future, it can pass a *view* of the kwargs dictionary *if* the callee would receive the exact same keys (or maybe even more generally), and then only do a deeper copy of the item pointers and order array if either the view or the original is changed. How would this work? Once again, what you receive in your extra keywords parameter is a mapping that contains all the keyword arguments that weren't matched to normal parameters and all the items of the unpacked mapping (if any) that weren't matched, but none of the keywords or mapping keys that were matched. A view into the unpacked mapping doesn't help unless you can somehow shadow it to hide some keys (the ones that matched parameters) and add others (the unmatched keywords). In fact, to get the ordered mapping we're looking for, you have to be able to add them _before_ the unpacked keys, but still preserving their own order. > > Anyway, even if things did work this way, one of Guido's objections is that he writes lots of code that keeps the kwargs dict around and adds to it later. (Presumably he has a method that takes **kw and stores it in an instance attribute, although he didn't give examples.) Using something which is not a dict, but only fakes it just well enough for passing keyword arguments, surely isn't going to satisfy him. > > Well, what exactly do you mean "not a dict"? It won't be a literal dict type, no, but it might be a subclass. It would be better if the InternedDict could be as fast as possible (for the usual case), but I don't think that it would absolutely cripple the idea to make it Mutable If it's just a subclass without actually being a subtype (meaning it can do anything a dict can do, including adding arbitrary keys), then all this does is trick Guido's code and prevent it from even having a way to test or document its preconditions. (This is what the Liskov Substitution Principle is all about--and it's even more important in duck-typed languages than static-typed, because isinstance in Python is used only to check subtitling, unlike dynamic_cast in C++.) And if it actually is a subtype, I don't see how you're going to accelerate it; it it can assign to keys of any type, then it can't assume all its keys are strings. > ... A-and on the flip side, I put forth that maybe Guido shouldn't be doing that kind of thing in the first place (*gasp*)! If he wanted a copy, he should've explicitly copied it with dict(), instead of relying on behavior that I've never even seen a doc reference for! I actually agree here. This is my argument against the OrderedDict kwargs PEP having a decorator that gives you a dict instead of an OrderedDict: it's just as easy, and clearer and more explicit, for Guido to save dict(kwargs) as for him to add @unordered_kwargs to the function. > (Not sure if I actually believe that **kwargs should be some subclass of FrozenDict, but maybe someone else does, and they can convince us.) > > > For example, there's plenty of Python 2.x code that fakes keyword-only arguments by calling kwargs.pop; that code still runs in Python 3.4, and I don't think you want to break it in 3.5. So you're effectively going to have to build a full MutableMapping. > > So... you're saying there's a chance we can break it in 3.6, right? No reason not to look to the future if backwards compatibility is what's keeping it from happening now. Well, I think it would have to go through the normal 3-version deprecation cycle, not just be dropped in to the next version. But, more importantly, a change that breaks backward compatibility takes a much more compelling argument than one that doesn't. It looks like everyone believes that the simpler OrderedDict proposal (using the already-planned C implementation of OrderedDict and changing nothing else about the code except to copy an unpacked mapping items in iteration order after the keywords rather than before) will probably be fast enough and small enough except in certain edge cases. A more complicated alternative proposal that's also probably fast enough and small enough except in those edge cases, but breaks those edge cases and others as well in a backward-incompatible way, doesn't seem like a win. > > You really think so? > > Yes, I believe that it is not impossible that someone knows how to make pointer lookups faster than general hash lookups (a function call) of general dicts. This may be because I haven't looked at the dict implementation, or because the dict implementation could be better (see https://mail.python.org/pipermail/python-dev/2012-December/123028.html), but I'm putting forth the idea because maybe it's not impossible, and someone can make it possible. The post you're referring to doesn't make lookup any faster; it makes most dicts smaller, and it makes iteration faster, but it has no effect on lookup. Also, notice that it explicitly says it has no effect on the existing optimization for lookup in string-only dicts. So, if you're hoping that it could somehow make string-only dicts faster, you need to read it again. > > But, if it is, hashing the strings to put them into an interned string table, and then hashing those interned pointers to put into the mini-dict, isn't going to be faster. > > Why not? If an InternedDict is passing args to a newly-created IDict, the hashing could be a single instruction: modulo by 63 (or something). (And then maybe multiple instructions to crawl through a linked list.) And knowing you don't HAVE to call a str.hash is surely better than calling even a cached str.hash, no? But you have to hash the keywords in the first place, or do something equivalent, to intern them. Unless you intend to intern another copy of the same strings on every function call? Also, you have to keep in mind that, on top of any lookups the function body does, the function call machinery also has to effectively look up every parameter and (to catch duplicate-keyword errors) every keyword argument in the unpacked mapping. If those lookups weren't fast enough, then your change doesn't help. If they were, then your change is probably unnecessary. > Point is, again, I don't know that it won't be faster. You seem to know that it won't be faster, but I don't see you saying that there CAN'T be such cleverness when kwargs are nice and forwarded nicely, even with deletions or additions allowed. Again, kwargs parameters aren't forwarded from anything, they're built from the unmatched keyword arguments, both normal and unpacked. And unpacking arguments aren't forwarded to anything, they're unpacked into keyword arguments, some of which may be gathered into a kwargs parameter. If you're only trying to optimize the case of forwarding for delegation, then your idea makes more sense, but I don't think that's the case anyone is looking to optimize. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Apr 15 21:02:48 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 15 Apr 2014 12:02:48 -0700 Subject: [Python-ideas] Literals call local constructors()? In-Reply-To: References: Message-ID: On Apr 15, 2014, at 10:24, "Franklin? Lee" wrote: > I think C Anthony is saying that he tried to make it happen, and it didn't, but he figured something else out, which is what is relevant to him here. What's the something else? It seems to me he was asking why hacking dict literals doesn't work (either from an implementation point of view or from a language design point of view) because he _hadn't_ figured it out. And that's what Chris Angelico answered (from both points of view). > I am mostly sure he isn't arguing for the ability to hack dict literals > On Tue, Apr 15, 2014 at 6:40 AM, Chris Angelico wrote: >> On Tue, Apr 15, 2014 at 8:11 PM, C Anthony Risinger wrote: >> > This not working is unexpected, if considering the expectation literals: >> > >> > {..} and [...] >> > >> > ...translate to: >> > >> > dict(...) and list(...) [int(..) and str(..)] >> > >> > Why isn't/can't this be true? >> >> They don't. They translate into dict-creation bytecodes. In CPython: >> >> Python 3.5.0a0 (default:6a0def54c63d, Mar 26 2014, 01:11:09) >> [GCC 4.7.2] on linux >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import dis >> >>> def make_dict1(): >> ... return {'hello':1, 'world':2} >> ... >> >>> def make_dict2(): >> ... return dict(hello=1, world=2) >> ... >> >>> dis.dis(make_dict1) >> 2 0 BUILD_MAP 2 >> 3 LOAD_CONST 1 (1) >> 6 LOAD_CONST 2 ('hello') >> 9 STORE_MAP >> 10 LOAD_CONST 3 (2) >> 13 LOAD_CONST 4 ('world') >> 16 STORE_MAP >> 17 RETURN_VALUE >> >>> dis.dis(make_dict2) >> 2 0 LOAD_GLOBAL 0 (dict) >> 3 LOAD_CONST 1 ('hello') >> 6 LOAD_CONST 2 (1) >> 9 LOAD_CONST 3 ('world') >> 12 LOAD_CONST 4 (2) >> 15 CALL_FUNCTION 512 (0 positional, 2 keyword pair) >> 18 RETURN_VALUE >> >> If you want the name dict to be looked up, look up the name dict. >> Otherwise, code like this would be very VERY confusing: >> >> >>> def make_dict3(): >> ... dict = {'hello':1, 'world':2} >> ... return dict >> ... >> >>> dis.dis(make_dict3) >> 2 0 BUILD_MAP 2 >> 3 LOAD_CONST 1 (1) >> 6 LOAD_CONST 2 ('hello') >> 9 STORE_MAP >> 10 LOAD_CONST 3 (2) >> 13 LOAD_CONST 4 ('world') >> 16 STORE_MAP >> 17 STORE_FAST 0 (dict) >> >> 3 20 LOAD_FAST 0 (dict) >> 23 RETURN_VALUE >> >> If that had to actually call dict(), it would raise UnboundLocalError >> for something that doesn't seem to reference locals before assigning >> to them. As it is, it's exactly the same as make_dict1 except that it >> does a store/load unnecessarily. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Apr 15 21:25:46 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 15 Apr 2014 12:25:46 -0700 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: Sent from a random iPhone On Apr 14, 2014, at 21:41, Terry Reedy wrote: > As I said in response to this same post on python-list, one problem with mechanical semantic interpretation of version numbers, the supposed justification for the system, is that such interpretation does not work as soon as one accidentally makes a release that violates the semantic rules. To be fair, semver tries to deal with that. It's arguable whether it does so sufficiently, but at least it doesn't ignore the problem. Also, having a semantic interpretation that's almost always mechanically interpretable but occasionally not isn't completely useless. If my project relies on 2.3.4+ but not 3.x of your project, and your 2.3.7 breaks backward compatibility, that's a bug in your project. As long as you publicize the bug and any appropriate workarounds and fix it (e.g., by releasing a 2.3.8), it's the same as any other bug (like your 2.3.7 failing to install, or crashing any time anyone calls the spam function). If, say, RedHat or Apple released a distribution with your 2.3.7, it would be just as bad if the version number was broken as if the code was broken, not worse. From yoavglazner at gmail.com Tue Apr 15 21:57:59 2014 From: yoavglazner at gmail.com (yoav glazner) Date: Tue, 15 Apr 2014 22:57:59 +0300 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> Message-ID: On Apr 15, 2014 9:13 PM, "Franklin? Lee" wrote: > > Well, a little effort on the last ditch. > > Like I said before, I don't see the need for modification to mean that this can't be made to work. I still think it would be possible to make the InternDict start off with the knowledge of the keys, and then upon attempt to add, it can either: > > - use a second internal array to store non-str keys (or just non-interned keys? or just all new keys?), and doing a check to see, "Am I trying to access a str key?". The hash would have an index into one of the two arrays. If it's storing new str keys in this new array (as opposed to only non-str keys), the interpreter would flatten again upon **blowup. > > (Example indexing for the second array: -1 is the first NEW element, then -2... If you ask for an element, and it's a str, then intern it and hash its interned ID and look it up on the hash table. If it's negative, it's in the second array. If you ask for the ordered keys, it will trek through the original array forwards, then the newarray backwards. Also, the standard case doesn't need to worry about lookup overhead, because it only looks for a second array if the hash table reports a negative index. It does need to check the second-array pointer if it's getting the elements in order.) > > - internally convert to a regular dict. This is undesirable, because it would lose order, which would be surprising to users. > > and upon delete, it would have to deal with deleting two entries (hash table and array). Fortunately, it wouldn't need to shift everything down, probably, and the unpacking call can just get the elements in sorted order. > > Both of these, alas, would cause additional overhead on add/delete, and the first causes extra bookkeeping even without that use (a pointer to a second internal array, even if that pointer is null, and occasional checking of that pointer when working with the whole table; and always intern strings going in). And if the kwargs are an OrderedDict, and not just acting like an ordered dict until you change it, it would have to remember the order of those new members, even if they're not going to be repacked into kwargs, and even if the user doesn't care about the order. > > But I don't know enough to say that that's a killer. As in, maybe it improves the usual case (calls) enough that the harm to the modifying case is cancelled out for big programs. Maybe non-modifying-case speedup + modifying-case slowdown < 0, and maybe if it looks enough like a dict, no user ever has to know. That's about five maybes, for those not keeping track. > > > On Tue, Apr 15, 2014 at 9:01 AM, Eric Snow wrote: >> >> On Mon, Apr 14, 2014 at 10:12 PM, Franklin? Lee >> wrote: >> > Point is, again, I don't know that it won't be faster. You seem to know that >> > it won't be faster, but I don't see you saying that there CAN'T be such >> > cleverness when kwargs are nice and forwarded nicely, even with deletions or >> > additions allowed. >> >> The point is moot. The object into which uncollected kwargs is bound >> must be a complete mapping instance that supports any type of key. We >> certainly *initialize* it with string keys, but once it is bound in >> the function, code is free to do anything that works with a normal >> dict (think Liskov). Limiting capability is a non-starter. >> >> That said, thanks for posting the idea. It wasn't all that unfeasible >> (unlike other ideas that land here ), even if it ultimately >> wasn't acceptable. You can also "upgrade" the interned dict upon addition Kwargs.__class__ = dict Only in C and be careful ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Tue Apr 15 22:09:40 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 15 Apr 2014 16:09:40 -0400 Subject: [Python-ideas] Literals call local constructors()? In-Reply-To: References: Message-ID: Nope, I retract what I said. On Tue, Apr 15, 2014 at 3:02 PM, Andrew Barnert wrote: > On Apr 15, 2014, at 10:24, "Franklin? Lee" > wrote: > > I think C Anthony is saying that he tried to make it happen, and it > didn't, but he figured something else out, which is what is relevant to him > here. > > > What's the something else? > > It seems to me he was asking why hacking dict literals doesn't work > (either from an implementation point of view or from a language design > point of view) because he _hadn't_ figured it out. And that's what Chris > Angelico answered (from both points of view). > > I am mostly sure he isn't arguing for the ability to hack dict literals > > > On Tue, Apr 15, 2014 at 6:40 AM, Chris Angelico wrote: > >> On Tue, Apr 15, 2014 at 8:11 PM, C Anthony Risinger >> wrote: >> > This not working is unexpected, if considering the expectation literals: >> > >> > {..} and [...] >> > >> > ...translate to: >> > >> > dict(...) and list(...) [int(..) and str(..)] >> > >> > Why isn't/can't this be true? >> >> They don't. They translate into dict-creation bytecodes. In CPython: >> >> Python 3.5.0a0 (default:6a0def54c63d, Mar 26 2014, 01:11:09) >> [GCC 4.7.2] on linux >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import dis >> >>> def make_dict1(): >> ... return {'hello':1, 'world':2} >> ... >> >>> def make_dict2(): >> ... return dict(hello=1, world=2) >> ... >> >>> dis.dis(make_dict1) >> 2 0 BUILD_MAP 2 >> 3 LOAD_CONST 1 (1) >> 6 LOAD_CONST 2 ('hello') >> 9 STORE_MAP >> 10 LOAD_CONST 3 (2) >> 13 LOAD_CONST 4 ('world') >> 16 STORE_MAP >> 17 RETURN_VALUE >> >>> dis.dis(make_dict2) >> 2 0 LOAD_GLOBAL 0 (dict) >> 3 LOAD_CONST 1 ('hello') >> 6 LOAD_CONST 2 (1) >> 9 LOAD_CONST 3 ('world') >> 12 LOAD_CONST 4 (2) >> 15 CALL_FUNCTION 512 (0 positional, 2 keyword pair) >> 18 RETURN_VALUE >> >> If you want the name dict to be looked up, look up the name dict. >> Otherwise, code like this would be very VERY confusing: >> >> >>> def make_dict3(): >> ... dict = {'hello':1, 'world':2} >> ... return dict >> ... >> >>> dis.dis(make_dict3) >> 2 0 BUILD_MAP 2 >> 3 LOAD_CONST 1 (1) >> 6 LOAD_CONST 2 ('hello') >> 9 STORE_MAP >> 10 LOAD_CONST 3 (2) >> 13 LOAD_CONST 4 ('world') >> 16 STORE_MAP >> 17 STORE_FAST 0 (dict) >> >> 3 20 LOAD_FAST 0 (dict) >> 23 RETURN_VALUE >> >> If that had to actually call dict(), it would raise UnboundLocalError >> for something that doesn't seem to reference locals before assigning >> to them. As it is, it's exactly the same as make_dict1 except that it >> does a store/load unnecessarily. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Apr 15 22:26:15 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 15 Apr 2014 16:26:15 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: On 4/15/2014 3:25 PM, Andrew Barnert wrote: > Sent from a random iPhone > > On Apr 14, 2014, at 21:41, Terry Reedy > wrote: > >> As I said in response to this same post on python-list, one problem >> with mechanical semantic interpretation of version numbers, the >> supposed justification for the system, is that such interpretation >> does not work as soon as one accidentally makes a release that >> violates the semantic rules. > > To be fair, semver tries to deal with that. It's arguable whether it > does so sufficiently, but at least it doesn't ignore the problem. In my original response on python-list, I quoted parts of the semver faq. I should have quoted instead of summarizing my original response. > Also, having a semantic interpretation that's almost always > mechanically interpretable but occasionally not isn't completely > useless. Indeed, I said that it is quite useful, but not perfect. A separate criticism of 'Python should officially support Semantic Versioning', interpreted as 'Python should use Semantic Versioning for its version numbers', is that a) Python already does (or at least tries to) with respect to micro/patch releases, but that the semver semantics for major/minor numbers are not practical for CPython as a bundle of perhaps 200 modules. But if Hees is not proposing a change in CPython numbering, then this is not relevant to what he did mean. > If my project relies on 2.3.4+ but not 3.x of your project, > and your 2.3.7 breaks backward compatibility, that's a bug in your > project. Agreed. But it means that one cannot use the simplistic algorithm given on the web page: usable = '2.3.4' <= version < '3.0.0'. > As long as you publicize the bug and any appropriate > workarounds and fix it (e.g., by releasing a 2.3.8), it's the same as > any other bug (like your 2.3.7 failing to install, or crashing any > time anyone calls the spam function). My point was and is that one needs a special case modification to the usable expression: usable = '2.3.4' <= version < '3.0.0' and version != '2.3.7' One semver rule is that a release cannot be modified, even to fix it, without modifying the version number. (This is another rule CPython follows.) This is a practical rule since once something is released on the internet, it cannot be deleted or edited everywhere, and it would be confusing to have different files with the same identifier. But not allowing edits does conflict with the usability rule. A partial* solution would be for PyPI to maintain a set containing 'denounced by the author' releases. CPython 3.0.* could go in such a set. Some releases that were replaced within a month could definitely go in such. Then a more generic usability expression could be usable = '2.3.4' <= version < '3.0.0' and version not in denounced and similarly for other dependency calculations. * partial because mistakes are not always noticed -- Terry Jan Reedy From ryan at ryanhiebert.com Tue Apr 15 22:51:14 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Tue, 15 Apr 2014 15:51:14 -0500 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: On Tue, Apr 15, 2014 at 3:26 PM, Terry Reedy wrote: > On 4/15/2014 3:25 PM, Andrew Barnert wrote: > >> >> If my project relies on 2.3.4+ but not 3.x of your project, >> and your 2.3.7 breaks backward compatibility, that's a bug in your >> project. >> > > Agreed. But it means that one cannot use the simplistic algorithm given on > the web page: usable = '2.3.4' <= version < '3.0.0'. > > You can if the author releases a 2.3.8 that un-breaks backward compatibility. In general, the packaging tools should install the latest version that meets all the requirements, so as long as no dependencies require the exact broken one to be used, it'll work. Of course, you _could_ still add that extra guard, it just isn't needed because pip will take care of it for you. Denouncement might also still be useful, but in the same vein, probably not needed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Apr 15 23:14:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Apr 2014 17:14:20 -0400 Subject: [Python-ideas] module version number support for semver.org In-Reply-To: References: Message-ID: On 15 April 2014 16:51, Ryan Hiebert wrote: > On Tue, Apr 15, 2014 at 3:26 PM, Terry Reedy wrote: >> >> On 4/15/2014 3:25 PM, Andrew Barnert wrote: >>> >>> >>> If my project relies on 2.3.4+ but not 3.x of your project, >>> and your 2.3.7 breaks backward compatibility, that's a bug in your >>> project. >> >> >> Agreed. But it means that one cannot use the simplistic algorithm given on >> the web page: usable = '2.3.4' <= version < '3.0.0'. >> > You can if the author releases a 2.3.8 that un-breaks backward > compatibility. In general, the packaging tools should install the latest > version that meets all the requirements, so as long as no dependencies > require the exact broken one to be used, it'll work. Of course, you _could_ > still add that extra guard, it just isn't needed because pip will take care > of it for you. Denouncement might also still be useful, but in the same > vein, probably not needed. For Python, please keep in mind that PEP 440 already covers this in detail: http://www.python.org/dev/peps/pep-0440/#version-specifiers It uses the semantic parts of semantic versioning, deals with date based versioning schemes cleanly, and includes all the flexibility needed to exclude known broken releases. The only part of semver that we don't allow is the unorderable suffixes, and those won't be added because we consider deterministic ordering to be a requirement. Regards, Nick. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From leewangzhong+python at gmail.com Tue Apr 15 23:32:20 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 15 Apr 2014 17:32:20 -0400 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: <48E80F4E-8EC0-4A69-950B-32990797EC3C@yahoo.com> References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> <48E80F4E-8EC0-4A69-950B-32990797EC3C@yahoo.com> Message-ID: > > but obviously wouldn't work if you tried to create the same type for > kwargs. > Sorry, I said that completely wrong. I mean that if it's already a special dict (one that can be optimized, for some definition of "can"), the interpreter can use a special algorithm for copying from such a dict. Otherwise, it'd just have to do regular iteration. More importantly, I think you're still missing the point that keyword > unpacking and extra-keyword parameters are not directly connected. > Rather than missing the point, I don't see any reason why they shouldn't be made connected. For example, how is this going to preserve keyword order in cases where > nobody unpacks a mapping along with the keywords? > (and) > Again, kwargs parameters aren't forwarded from anything, they're built > from the unmatched keyword arguments, both normal and unpacked. > An algorithm-ish: At callpoint, the caller would put explicit keywords into a new kwdict, then look for **somethingToUnpack. If it finds it, and it's a special dict, it packs "smartly". Otherwise, it iterates through the pack and adds it to the kwdict. The callee gets the kwdict and pops the entries that it makes explicit (allowing deletion isn't the biggest problem). The leftover kwdict is named kwargs or whatever. How would this work? > "it can pass a *view* of the kwargs dictionary *if* the callee would receive the exact same keys" Your counter is that it wouldn't work if the callee doesn't receive the exact same keys. In that case, it would do the more general thing I stated above, whereby it creates a new dict like it does currently, but in a "smarter" way. It's a separate idea. then all this does is trick Guido's code and prevent it from even having a > way to test or document its preconditions. I didn't understand this. What preconditions? And if it actually is a subtype, I don't see how you're going to accelerate > it; it it can assign to keys of any type, then it can't assume all its keys > are strings. > What I suggested was a second array of k-v pairs for newly-added items, still in order, but with more general keys. If done with ALL new keys, this might slow down lookup due to looking up the ID (in case it's interned) and then hashing and looking up the string itself. But maybe this kind of case (modifying incoming kwargs) is uncommon enough (as in, doesn't happen much relative to the general case) that the overall program is faster. I don't know. Again, I'm putting forth the idea for someone who knows better about the internals to say for sure. Also, even if this worked, you're putting the burden of ordering the > keywords on the caller, rather than on the function that actually cares > about the keyword order. That's the wrong place to put it. You've solved > the case of forwarding delegate methods, partial, etc., but not the > everyday case. > The everyday case only cares if the actual performance goes down. I don't see a philosophical reason to ban the possibility of caller-burden altogether. That's why I'm asking the question: if the ** mechanic is clever by using the restraints about what is legal to pack/unpack, would it improve performance to the point that order can be thrown in and overall performance is STILL better than the status quo? A more complicated alternative proposal that's also probably fast enough > and small enough except in those edge cases, but breaks those edge cases > and others as well in a backward-incompatible way, doesn't seem like a win. > It may be confusing, but I'm proposing multiple exclusive possibilities. - Break those edge cases, because I'm not convinced they should be allowed. - Don't break those edge cases, but it's okay to slow them down because everything else will be faster and make up for them. Breaking the edge cases isn't integral, but it might be desirable for the language, far off into the future. And I'm not so much fighting for this to happen (instead of the C OrderedDict) as trying to flesh out the idea, because I don't *know enough* to say that it *won't* be better. The post you're referring to doesn't make lookup any faster; it makes most > dicts smaller, and it makes iteration faster, but it has no effect on > lookup. Also, notice that it explicitly says it has no effect on the > existing optimization for lookup in string-only dicts. So, if you're hoping > that it could somehow make string-only dicts faster, you need to read it > again. > No. That line you quoted was about why my proposal WOULDN'T help. What I said was that I thought it could be better than the current use of dict, and it's possible that it only has that potential because dict isn't as good as it can be. In other words, maybe it CAN be better than the current implementation with dict, but instead of putting in the effort to make this, put the effort into making dict better. Note, though, that 1. Raymond Hettinger's dict changes WOULD make it ordered, giving us ordered kwargs as a byproduct. 2. It makes iteration faster, which, as far as I know, should make kwargs packing and unpacking faster. But rereading it, I see that it means dicts already have a str-only optimization. Rather than, "This dict can't be used for your suggested kwargs optimization," it's, "The current dict is probably already doing what you're suggesting, so it wouldn't be much of a performance improvement." But you have to hash the keywords in the first place, or do something > equivalent, to intern them. > Yes. It'd be hashed once per keyword per function, and once per explicit keyword in a call (so if a line calls multiple times, it would have to hash that, I suppose, until the interpreter gets smarter, or if it's already smart about it), and once per keyword in an expansion from a regular dict. Also, you have to keep in mind that, on top of any lookups the function > body does, the function call machinery also has to effectively look up > every parameter and (to catch duplicate-keyword errors) every keyword > argument in the unpacked mapping. If those lookups weren't fast enough, > then your change doesn't help. If they were, then your change is probably > unnecessary. > I don't see the relevance of needing to look up every parameter. What is the objection? -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Tue Apr 15 23:35:16 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Tue, 15 Apr 2014 17:35:16 -0400 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> <48E80F4E-8EC0-4A69-950B-32990797EC3C@yahoo.com> Message-ID: Correction: "Raymond Hettinger's dict changes WOULD make it ordered, giving us ordered kwargs as a byproduct." ? This is false. I forgot to remove this. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Apr 16 04:12:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Apr 2014 22:12:34 -0400 Subject: [Python-ideas] py launcher for Linux? Message-ID: Something that came up during PyCon was the idea of a "py" script that brought "Python Launcher for Windows" explicit version dispatch to Linux. Anyone care to try their hand at writing such a script? Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Apr 16 04:30:39 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 15 Apr 2014 19:30:39 -0700 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: Message-ID: Not me, but I would recommend giving it a more prosaic (and longer) name. When discussing this at PyCon I was initially confused by the multiple uses of "py" in the same sentence. :) On Tue, Apr 15, 2014 at 7:12 PM, Nick Coghlan wrote: > Something that came up during PyCon was the idea of a "py" script that > brought "Python Launcher for Windows" explicit version dispatch to Linux. > > Anyone care to try their hand at writing such a script? > > Cheers, > Nick. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Apr 16 05:59:37 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 15 Apr 2014 20:59:37 -0700 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> <48E80F4E-8EC0-4A69-950B-32990797EC3C@yahoo.com> Message-ID: On Apr 15, 2014, at 14:32, "Franklin? Lee" wrote: >> but obviously wouldn't work if you tried to create the same type for kwargs. > > Sorry, I said that completely wrong. I mean that if it's already a special dict (one that can be optimized, for some definition of "can"), the interpreter can use a special algorithm for copying from such a dict. Otherwise, it'd just have to do regular iteration. > >> More importantly, I think you're still missing the point that keyword unpacking and extra-keyword parameters are not directly connected. > > Rather than missing the point, I don't see any reason why they shouldn't be made connected. Because they're not conceptually connected. Just because people often get confused and think they're a matched pair doesn't mean the language should change to confuse people even further. >> For example, how is this going to preserve keyword order in cases where nobody unpacks a mapping along with the keywords? > > (and) > >> Again, kwargs parameters aren't forwarded from anything, they're built from the unmatched keyword arguments, both normal and unpacked. > > An algorithm-ish: At callpoint, the caller would put explicit keywords into a new kwdict, then look for **somethingToUnpack. If it finds it, and it's a special dict, it packs "smartly". Otherwise, it iterates through the pack and adds it to the kwdict. By 'packs "smartly"' you mean it copies the dict it's already started building into a new special dict so that it can then update that new special dict with the unpacking special dict? I suspect the cost of building a normal dict just to copy it to a special dict is going to cost a lot more than whatever the savings for your special dict. Anyway, you really should look at how CPython and other implementations do it before proposing changes. In CPython, the calling side puts keyword args on the stack, and also puts the unpacking dict on the stack, and then passes a count of keyword args and a flag indicating the presence of that dict to the callee. The callee then pops those, builds the new dict (which includes looking up each keyword in the dict to make sure it isn't a duplicate), fills in the param values, and fills the kwargs dict with what's left over. What you're proposing isn't a modification of that, but a completely different design. Which might be worth doing, but you either need to argue that it's better, or work with the existing design. > The callee gets the kwdict and pops the entries that it makes explicit (allowing deletion isn't the biggest problem). The leftover kwdict is named kwargs or whatever. > >> How would this work? > > "it can pass a *view* of the kwargs dictionary *if* the callee would receive the exact same keys" > > Your counter is that it wouldn't work if the callee doesn't receive the exact same keys. In that case, it would do the more general thing I stated above, whereby it creates a new dict like it does currently, but in a "smarter" way. It's a separate idea. > >> then all this does is trick Guido's code and prevent it from even having a way to test or document its preconditions. > > I didn't understand this. What preconditions? The precondition that it be something that acts like an actual dict (not just a MutableMapping), which you spell isinstance(it, dict). Taking something that isn't a dict an making it a subclass of dict doesn't fix the problem, it just means you can no longer test for the problem even if you want to. >> And if it actually is a subtype, I don't see how you're going to accelerate it; it it can assign to keys of any type, then it can't assume all its keys are strings. > > What I suggested was a second array of k-v pairs for newly-added items, still in order, but with more general keys. If done with ALL new keys, this might slow down lookup due to looking up the ID (in case it's interned) and then hashing and looking up the string itself. But maybe this kind of case (modifying incoming kwargs) is uncommon enough (as in, doesn't happen much relative to the general case) that the overall program is faster. I don't know. Again, I'm putting forth the idea for someone who knows better about the internals to say for sure. > >> Also, even if this worked, you're putting the burden of ordering the keywords on the caller, rather than on the function that actually cares about the keyword order. That's the wrong place to put it. You've solved the case of forwarding delegate methods, partial, etc., but not the everyday case. > > > The everyday case only cares if the actual performance goes down. I don't see a philosophical reason to ban the possibility of caller-burden altogether. That's why I'm asking the question: if the ** mechanic is clever by using the restraints about what is legal to pack/unpack, would it improve performance to the point that order can be thrown in and overall performance is STILL better than the status quo? > >> A more complicated alternative proposal that's also probably fast enough and small enough except in those edge cases, but breaks those edge cases and others as well in a backward-incompatible way, doesn't seem like a win. > > It may be confusing, but I'm proposing multiple exclusive possibilities. > - Break those edge cases, because I'm not convinced they should be allowed. > - Don't break those edge cases, but it's okay to slow them down because everything else will be faster and make up for them. > > Breaking the edge cases isn't integral, but it might be desirable for the language, far off into the future. > > And I'm not so much fighting for this to happen (instead of the C OrderedDict) as trying to flesh out the idea, because I don't *know enough* to say that it *won't* be better. > >> The post you're referring to doesn't make lookup any faster; it makes most dicts smaller, and it makes iteration faster, but it has no effect on lookup. Also, notice that it explicitly says it has no effect on the existing optimization for lookup in string-only dicts. So, if you're hoping that it could somehow make string-only dicts faster, you need to read it again. > > No. That line you quoted was about why my proposal WOULDN'T help. What I said was that I thought it could be better than the current use of dict, and it's possible that it only has that potential because dict isn't as good as it can be. In other words, maybe it CAN be better than the current implementation with dict, but instead of putting in the effort to make this, put the effort into making dict better. > > Note, though, that > 1. Raymond Hettinger's dict changes WOULD make it ordered, giving us ordered kwargs as a byproduct. > 2. It makes iteration faster, which, as far as I know, should make kwargs packing and unpacking faster. > > But rereading it, I see that it means dicts already have a str-only optimization. Rather than, "This dict can't be used for your suggested kwargs optimization," it's, "The current dict is probably already doing what you're suggesting, so it wouldn't be much of a performance improvement." > >> But you have to hash the keywords in the first place, or do something equivalent, to intern them. > > Yes. It'd be hashed once per keyword per function, and once per explicit keyword in a call (so if a line calls multiple times, it would have to hash that, I suppose, until the interpreter gets smarter, or if it's already smart about it), and once per keyword in an expansion from a regular dict. >> Also, you have to keep in mind that, on top of any lookups the function body does, the function call machinery also has to effectively look up every parameter and (to catch duplicate-keyword errors) every keyword argument in the unpacked mapping. If those lookups weren't fast enough, then your change doesn't help. If they were, then your change is probably unnecessary. > > I don't see the relevance of needing to look up every parameter. What is the objection? The fact that each keyword involves multiple lookups and only one iteration, and you're only speeding up iteration, means you're probably optimizing the part that doesn't matter. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Apr 16 10:07:25 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 16 Apr 2014 11:07:25 +0300 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: Message-ID: 16.04.14 05:12, Nick Coghlan ???????(??): > Something that came up during PyCon was the idea of a "py" script that > brought "Python Launcher for Windows" explicit version dispatch to Linux. > > Anyone care to try their hand at writing such a script? $ alias py= or script: $ cat py #!/bin/sh "$@" From leewangzhong+python at gmail.com Wed Apr 16 10:25:43 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 16 Apr 2014 04:25:43 -0400 Subject: [Python-ideas] Preserving **kwargs order (was: Re: OrderedDict literals) In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> <48E80F4E-8EC0-4A69-950B-32990797EC3C@yahoo.com> Message-ID: > Anyway, you really should look at how CPython and other implementations do it before proposing changes. And here's the key point: I wasn't trying to propose a change. I was trying to offer a suggestion (taking advantage of the keyspace), which someone else could decide was worth looking into. I did not come here intending to fight for the idea, but you have put me on the defensive, and I've felt the need to clarify what I was imagining as something more than "just wrong", or "missing the point", since we have different images of what I'm imagining. I'm not upset, but you seem to have an implicit assumption that this is what I WANT to have implemented, and it's currently not a very productive conversation (for Python, though I've learned in the course of it). -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Apr 16 12:19:00 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 16 Apr 2014 19:19:00 +0900 Subject: [Python-ideas] Topicality [was: Preserving **kwargs order] In-Reply-To: References: <1397531513.92210.YahooMailNeo@web181002.mail.ne1.yahoo.com> <48E80F4E-8EC0-4A69-950B-32990797EC3C@yahoo.com> Message-ID: <21326.22804.267974.461890@uwakimon.sk.tsukuba.ac.jp> Franklin? Lee writes: > And here's the key point: I wasn't trying to propose a change. But that's what this list is for, proposing changes. Technically speaking, random "why not" ideas belong on python-list. Nobody's going to make a big deal of it, especially when you're injecting the idea into a related existing thread. *But* you should expect somebody assume you're actually proposing something, and to follow up by explaining "why not" in detail if you post such ideas here. If you don't really have anything invested in the idea, I would recommend just saying, "OK, and thank you for the detailed explanation. I learned a lot." (N.B. That's what you actually did end up saying!) From eric at trueblade.com Wed Apr 16 12:25:51 2014 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 16 Apr 2014 06:25:51 -0400 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: Message-ID: <534E5AAF.6080808@trueblade.com> On 04/15/2014 10:12 PM, Nick Coghlan wrote: > Something that came up during PyCon was the idea of a "py" script that > brought "Python Launcher for Windows" explicit version dispatch to Linux. > > Anyone care to try their hand at writing such a script? I meant to bring this up at the PyCon sprints. On Windows, the launcher finds the various versions of Python by looking in the registry. How is a Linux version supposed to find what versions of Python are installed? I have a pristine 2.7 in /usr/local/bin and the system python 2.7 in /usr/bin. I've also seen /opt/bin used, and other places deliberately not on PATH. How would a launcher find these, let alone decide which one to use? I'll be around the sprints for a very short time this morning, if anyone wants to discuss it. Eric. From leewangzhong+python at gmail.com Wed Apr 16 13:23:15 2014 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 16 Apr 2014 07:23:15 -0400 Subject: [Python-ideas] Topicality [was: Preserving **kwargs order] Message-ID: > But that's what this list is for, proposing changes. That's wordplay. The list is for proposing changes, but it's also for discussing them. I think making a suggestion (simply: using kw restraints to prevent regression) for how to implement proposed changes (kwargs as ordered dict) is not simply "not a big deal", but on-topic. My continued defense of it is because I don't agree with the reasons put forth for it being *impossible*. For example, the claims that it is *necessary* to allow loss of order upon addition to a kw-optimized dict. I myself gave a bunch of reasons why it MIGHT not be faster (biggest so far being "str-optimization may already be implemented"), but I'm not convinced that those reasons given by others can't be worked around. I'm not opposed to criticism of the idea. But I believe we're going to talk a lot more than necessary if people respond as if I'm convinced it WILL work, or that I am pushing for it to happen. I'm thinking of writing up pseudo-code for it later, so I can at least share a common reference point, and solidify the concept. Or I might find that the already-implemented str-dict optimizations can't be improved upon, and report back. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Apr 16 13:50:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 16 Apr 2014 07:50:17 -0400 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: <534E5AAF.6080808@trueblade.com> References: <534E5AAF.6080808@trueblade.com> Message-ID: On 16 Apr 2014 06:27, "Eric V. Smith" wrote: > > On 04/15/2014 10:12 PM, Nick Coghlan wrote: > > Something that came up during PyCon was the idea of a "py" script that > > brought "Python Launcher for Windows" explicit version dispatch to Linux. > > > > Anyone care to try their hand at writing such a script? > > I meant to bring this up at the PyCon sprints. On Windows, the launcher > finds the various versions of Python by looking in the registry. How is > a Linux version supposed to find what versions of Python are installed? > > I have a pristine 2.7 in /usr/local/bin and the system python 2.7 in > /usr/bin. I've also seen /opt/bin used, and other places deliberately > not on PATH. How would a launcher find these, let alone decide which one > to use? Note that my main interest here is in making commands like: py -3 -m pip install foo cross platform rather than Windows specific. At the moment we don't have a way to explicitly invoke Python 3 that works everywhere. Given the way POSIX installations of Python name things, simple string munging, along with delegation to the shell's normal shebang line processing when given a script without nominating a particular version to use, should suffice to largely reproduce the PEP 397 behaviour. The launcher config file also provides an opportunity to make the "default Python" a per user setting without altering which version the "python" symlink refers to. Cheers, Nick. > > I'll be around the sprints for a very short time this morning, if anyone > wants to discuss it. > > Eric. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Apr 16 15:08:43 2014 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 16 Apr 2014 09:08:43 -0400 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: <534E5AAF.6080808@trueblade.com> Message-ID: <534E80DB.5050601@trueblade.com> On 04/16/2014 07:50 AM, Nick Coghlan wrote: > > On 16 Apr 2014 06:27, "Eric V. Smith" > wrote: >> >> On 04/15/2014 10:12 PM, Nick Coghlan wrote: >> > Something that came up during PyCon was the idea of a "py" script that >> > brought "Python Launcher for Windows" explicit version dispatch to > Linux. >> > >> > Anyone care to try their hand at writing such a script? >> >> I meant to bring this up at the PyCon sprints. On Windows, the launcher >> finds the various versions of Python by looking in the registry. How is >> a Linux version supposed to find what versions of Python are installed? >> >> I have a pristine 2.7 in /usr/local/bin and the system python 2.7 in >> /usr/bin. I've also seen /opt/bin used, and other places deliberately >> not on PATH. How would a launcher find these, let alone decide which one >> to use? > > Note that my main interest here is in making commands like: > > py -3 -m pip install foo > > cross platform rather than Windows specific. At the moment we don't have > a way to explicitly invoke Python 3 that works everywhere. > > Given the way POSIX installations of Python name things, simple string > munging, along with delegation to the shell's normal shebang line > processing when given a script without nominating a particular version > to use, should suffice to largely reproduce the PEP 397 behaviour. > > The launcher config file also provides an opportunity to make the > "default Python" a per user setting without altering which version the > "python" symlink refers to. Okay. I'll take a crack at this. Eric. From solipsis at pitrou.net Wed Apr 16 16:43:01 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 16 Apr 2014 16:43:01 +0200 Subject: [Python-ideas] py launcher for Linux? References: <534E5AAF.6080808@trueblade.com> Message-ID: <20140416164301.683d5bee@fsol> On Wed, 16 Apr 2014 07:50:17 -0400 Nick Coghlan wrote: > cross platform rather than Windows specific. At the moment we don't have a > way to explicitly invoke Python 3 that works everywhere. How about "python3"? That's not different from "python" for 2.x. If we don't install anything named "python3.exe" under Windows, then perhaps we should, instead of adding a clunky wrapper under POSIX. Regards Antoine. From storchaka at gmail.com Wed Apr 16 17:18:39 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 16 Apr 2014 18:18:39 +0300 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: <534E5AAF.6080808@trueblade.com> References: <534E5AAF.6080808@trueblade.com> Message-ID: 16.04.14 13:25, Eric V. Smith ???????(??): > I meant to bring this up at the PyCon sprints. On Windows, the launcher > finds the various versions of Python by looking in the registry. How is > a Linux version supposed to find what versions of Python are installed? Debian-like systems use "alternatives". https://wiki.debian.org/DebianAlternatives From toddw at activestate.com Wed Apr 16 18:48:12 2014 From: toddw at activestate.com (Todd Whiteman) Date: Wed, 16 Apr 2014 09:48:12 -0700 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: Message-ID: <534EB44C.4060708@activestate.com> On 14-04-15 07:12 PM, Nick Coghlan wrote: > Something that came up during PyCon was the idea of a "py" script that > brought "Python Launcher for Windows" explicit version dispatch to Linux. > > Anyone care to try their hand at writing such a script? Do you mean something like "pyenv": https://github.com/yyuu/pyenv From yselivanov.ml at gmail.com Wed Apr 16 19:09:48 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 16 Apr 2014 13:09:48 -0400 Subject: [Python-ideas] optional parameters Message-ID: <534EB95C.8060006@gmail.com> Hello, There is a very common pattern for creating optional arguments when you can't use None: _optional = object() def foo(*, arg1='spam', arg3=None, arg4=_optional): if arg4 is _optional: # caller didn't pass *anything* for arg4 else: # caller did pass some (maybe None) value for arg4 It's a bit annoying to create this marker objects, and also, if you try to render a signature of such function, you'll get something like: "(*, arg1='spam', arg3=None, arg4=)" What if we add a standard marker for this use-case: functools.optional or inspect.Parameter.optional? Yury From stephen at xemacs.org Wed Apr 16 19:21:16 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 17 Apr 2014 02:21:16 +0900 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: <534E5AAF.6080808@trueblade.com> Message-ID: <21326.48140.503483.145304@uwakimon.sk.tsukuba.ac.jp> Serhiy Storchaka writes: > 16.04.14 13:25, Eric V. Smith ???????(??): > > I meant to bring this up at the PyCon sprints. On Windows, the launcher > > finds the various versions of Python by looking in the registry. How is > > a Linux version supposed to find what versions of Python are installed? > > Debian-like systems use "alternatives". > > https://wiki.debian.org/DebianAlternatives Eric's example of a "pristine" build in /usr/local/bin shows that we need to consider whether a python not known to the system PMS should be considered to be "installed". I would think on POSIX systems anything on the PATH should be considered to be "installed", but that's not an exact analogy to Windows semantics of "look in the registry". From phd at phdru.name Wed Apr 16 19:28:28 2014 From: phd at phdru.name (Oleg Broytman) Date: Wed, 16 Apr 2014 19:28:28 +0200 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: <21326.48140.503483.145304@uwakimon.sk.tsukuba.ac.jp> References: <534E5AAF.6080808@trueblade.com> <21326.48140.503483.145304@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140416172828.GA1120@phdru.name> On Thu, Apr 17, 2014 at 02:21:16AM +0900, "Stephen J. Turnbull" wrote: > Serhiy Storchaka writes: > > 16.04.14 13:25, Eric V. Smith ???????(??): > > > I meant to bring this up at the PyCon sprints. On Windows, the launcher > > > finds the various versions of Python by looking in the registry. How is > > > a Linux version supposed to find what versions of Python are installed? > > > > Debian-like systems use "alternatives". > > > > https://wiki.debian.org/DebianAlternatives > > Eric's example of a "pristine" build in /usr/local/bin shows that we > need to consider whether a python not known to the system PMS should > be considered to be "installed". I would think on POSIX systems > anything on the PATH should be considered to be "installed", but > that's not an exact analogy to Windows semantics of "look in the > registry". I believe there is no need to think about what's *installed*; it's enough to think about what's *available* -- in PATH and in the config file. Who cares from what sources and in what ways they became available? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From guido at python.org Wed Apr 16 19:29:07 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 16 Apr 2014 10:29:07 -0700 Subject: [Python-ideas] optional parameters In-Reply-To: <534EB95C.8060006@gmail.com> References: <534EB95C.8060006@gmail.com> Message-ID: TBH I think it's usually an anti-pattern when you have an API where the absence of a parameter is not equivalent to some default value. It makes wrapping such APIs awkward, e.g. def logging_foo(*, arg1='something_other_than_spam', arg3=None, arg4=): loging.info('calling foo(arg1=%r, arg3=%r, arg4=%r)', arg1, arg3, arg4) return foo(arg1=arg1, arg3=arg3, arg4=arg4) On Wed, Apr 16, 2014 at 10:09 AM, Yury Selivanov wrote: > Hello, > > There is a very common pattern for creating optional arguments > when you can't use None: > > _optional = object() > def foo(*, arg1='spam', arg3=None, arg4=_optional): > if arg4 is _optional: > # caller didn't pass *anything* for arg4 > else: > # caller did pass some (maybe None) value for arg4 > > It's a bit annoying to create this marker objects, and also, > if you try to render a signature of such function, you'll get > something like: > > "(*, arg1='spam', arg3=None, arg4=)" > > What if we add a standard marker for this use-case: > functools.optional or inspect.Parameter.optional? > > > Yury > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Wed Apr 16 19:59:37 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 16 Apr 2014 10:59:37 -0700 Subject: [Python-ideas] optional parameters In-Reply-To: References: <534EB95C.8060006@gmail.com> Message-ID: On Wed, Apr 16, 2014 at 10:29 AM, Guido van Rossum wrote: > TBH I think it's usually an anti-pattern when you have an API where the > absence of a parameter is not equivalent to some default value. It makes > wrapping such APIs awkward, e.g. > > def logging_foo(*, arg1='something_other_than_spam', arg3=None, > arg4=): > loging.info('calling foo(arg1=%r, arg3=%r, arg4=%r)', arg1, arg3, arg4) > return foo(arg1=arg1, arg3=arg3, arg4=arg4) Wrapping the APIs in such a tightly coupled way is already awkward, since if the argspec (e.g. the default) changes, so must your wrapper. *args/**kwargs are better when possible. -- Devin > On Wed, Apr 16, 2014 at 10:09 AM, Yury Selivanov > wrote: >> >> Hello, >> >> There is a very common pattern for creating optional arguments >> when you can't use None: >> >> _optional = object() >> def foo(*, arg1='spam', arg3=None, arg4=_optional): >> if arg4 is _optional: >> # caller didn't pass *anything* for arg4 >> else: >> # caller did pass some (maybe None) value for arg4 >> >> It's a bit annoying to create this marker objects, and also, >> if you try to render a signature of such function, you'll get >> something like: >> >> "(*, arg1='spam', arg3=None, arg4=)" >> >> What if we add a standard marker for this use-case: >> functools.optional or inspect.Parameter.optional? >> >> >> Yury >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From vernondcole at gmail.com Wed Apr 16 20:20:48 2014 From: vernondcole at gmail.com (Vernon D. Cole) Date: Wed, 16 Apr 2014 12:20:48 -0600 Subject: [Python-ideas] optional parameters Message-ID: On Wed, Apr 16, 2014 at 10:09 AM, Yury Selivanov wrote: > There is a very common pattern for creating optional arguments > when you can't use None: > > > > It's a bit annoying to create this marker objects, and also, > if you try to render a signature of such function, you'll get > something like: > > "(*, arg1='spam', arg3=None, arg4=)" > > What if we add a standard marker for this use-case: > functools.optional or inspect.Parameter.optional? > > There is already a singleton which works very well for this use case: def foo(*, arg1='spam', arg3=None, arg4=NotImplemented): if arg4 is NotImplemented: # caller didn't pass *anything* for arg4 else: # caller did pass some (maybe None) value for arg4 It is already defined, and reads like sensible English. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pyideas at gmail.com Wed Apr 16 20:22:29 2014 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Wed, 16 Apr 2014 13:22:29 -0500 Subject: [Python-ideas] optional parameters In-Reply-To: <534EB95C.8060006@gmail.com> References: <534EB95C.8060006@gmail.com> Message-ID: On Wed, Apr 16, 2014 at 12:09 PM, Yury Selivanov wrote: > Hello, > > There is a very common pattern for creating optional arguments > when you can't use None: > > _optional = object() > def foo(*, arg1='spam', arg3=None, arg4=_optional): > if arg4 is _optional: > # caller didn't pass *anything* for arg4 > else: > # caller did pass some (maybe None) value for arg4 > > It's a bit annoying to create this marker objects, and also, > if you try to render a signature of such function, you'll get > something like: > > "(*, arg1='spam', arg3=None, arg4=)" > > What if we add a standard marker for this use-case: > functools.optional or inspect.Parameter.optional? Has Ellipsis (...) been suggested and rejected for this use before? It's not an absolutely perfect fit, but it looks better to me: def foo(*, arg1='spam', arg3=None, arg4=...): if arg4 is Ellipsis: # caller didn't pass anything for arg4 ... else: # caller passed something, possibly None, for arg4 ... The signature would look like "(*, arg1='spam', arg3=None, arg4=Ellipsis)", unless inspect.Signature.__str__ special-cased Ellipsis to be "..." (or "" if we officially blessed Ellipsis for the purpose). -- Zach From guido at python.org Wed Apr 16 20:31:24 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 16 Apr 2014 11:31:24 -0700 Subject: [Python-ideas] optional parameters In-Reply-To: References: <534EB95C.8060006@gmail.com> Message-ID: This thread is derailing quickly. :-) Does anyone remember why the marker pattern was invented? (That's a rhetorical question, in case you wondered. :-) On Wed, Apr 16, 2014 at 11:22 AM, Zachary Ware < zachary.ware+pyideas at gmail.com> wrote: > On Wed, Apr 16, 2014 at 12:09 PM, Yury Selivanov > wrote: > > Hello, > > > > There is a very common pattern for creating optional arguments > > when you can't use None: > > > > _optional = object() > > def foo(*, arg1='spam', arg3=None, arg4=_optional): > > if arg4 is _optional: > > # caller didn't pass *anything* for arg4 > > else: > > # caller did pass some (maybe None) value for arg4 > > > > It's a bit annoying to create this marker objects, and also, > > if you try to render a signature of such function, you'll get > > something like: > > > > "(*, arg1='spam', arg3=None, arg4=)" > > > > What if we add a standard marker for this use-case: > > functools.optional or inspect.Parameter.optional? > > Has Ellipsis (...) been suggested and rejected for this use before? > It's not an absolutely perfect fit, but it looks better to me: > > def foo(*, arg1='spam', arg3=None, arg4=...): > if arg4 is Ellipsis: > # caller didn't pass anything for arg4 > ... > else: > # caller passed something, possibly None, for arg4 > ... > > The signature would look like "(*, arg1='spam', arg3=None, > arg4=Ellipsis)", unless inspect.Signature.__str__ special-cased > Ellipsis to be "..." (or "" if we officially blessed > Ellipsis for the purpose). > > -- > Zach > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Apr 16 20:39:07 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 16 Apr 2014 13:39:07 -0500 Subject: [Python-ideas] optional parameters In-Reply-To: References: <534EB95C.8060006@gmail.com> Message-ID: On Wed, Apr 16, 2014 at 12:29 PM, Guido van Rossum wrote: > TBH I think it's usually an anti-pattern when you have an API where the > absence of a parameter is not equivalent to some default value. It makes > wrapping such APIs awkward, e.g. > > def logging_foo(*, arg1='something_other_than_spam', arg3=None, > arg4=): > loging.info('calling foo(arg1=%r, arg3=%r, arg4=%r)', arg1, arg3, > arg4) > return foo(arg1=arg1, arg3=arg3, arg4=arg4) > > I've had to do that before when writing decorators that take parameters, e.g: def mydecorator(arg1,arg2=None): def _f(f): if arg2 is None: arg2 = f.something ... return f return _f Which sucks when None is a valid value. > > On Wed, Apr 16, 2014 at 10:09 AM, Yury Selivanov wrote: > >> Hello, >> >> There is a very common pattern for creating optional arguments >> when you can't use None: >> >> _optional = object() >> def foo(*, arg1='spam', arg3=None, arg4=_optional): >> if arg4 is _optional: >> # caller didn't pass *anything* for arg4 >> else: >> # caller did pass some (maybe None) value for arg4 >> >> It's a bit annoying to create this marker objects, and also, >> if you try to render a signature of such function, you'll get >> something like: >> >> "(*, arg1='spam', arg3=None, arg4=)" >> >> What if we add a standard marker for this use-case: >> functools.optional or inspect.Parameter.optional? >> >> >> Yury >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Apr 16 21:45:43 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 16 Apr 2014 15:45:43 -0400 Subject: [Python-ideas] optional parameters In-Reply-To: References: <534EB95C.8060006@gmail.com> Message-ID: <534EDDE7.8030204@gmail.com> On 2014-04-16, 1:29 PM, Guido van Rossum wrote: > TBH I think it's usually an anti-pattern when you have an API where the > absence of a parameter is not equivalent to some default value. It makes > wrapping such APIs awkward, e.g. > > def logging_foo(*, arg1='something_other_than_spam', arg3=None, > arg4=): > loging.info('calling foo(arg1=%r, arg3=%r, arg4=%r)', arg1, arg3, arg4) > return foo(arg1=arg1, arg3=arg3, arg4=arg4) Agree, wrapping is hard, but you can always use **kwargs. The pattern is sometimes useful for decorators (as Ryan Gonzalez shows in this thread). But basically, you need this when 'None' is a legitimate value for some optional argument, and you want users to pass explicitly. For instance, see inspect.Signature.replace method: http://hg.python.org/cpython/file/e4ee0b15cc4f/Lib/inspect.py#l2193 I'm not saying it's a very common thing to do, but sometimes you need this, and having the need of doing some ad hoc hacks is slightly annoying. Yury From g.rodola at gmail.com Wed Apr 16 21:55:48 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 16 Apr 2014 21:55:48 +0200 Subject: [Python-ideas] optional parameters In-Reply-To: <534EDDE7.8030204@gmail.com> References: <534EB95C.8060006@gmail.com> <534EDDE7.8030204@gmail.com> Message-ID: On Wed, Apr 16, 2014 at 9:45 PM, Yury Selivanov wrote: > > On 2014-04-16, 1:29 PM, Guido van Rossum wrote: > >> TBH I think it's usually an anti-pattern when you have an API where the >> absence of a parameter is not equivalent to some default value. It makes >> wrapping such APIs awkward, e.g. >> >> def logging_foo(*, arg1='something_other_than_spam', arg3=None, >> arg4=): >> loging.info('calling foo(arg1=%r, arg3=%r, arg4=%r)', arg1, arg3, >> arg4) >> return foo(arg1=arg1, arg3=arg3, arg4=arg4) >> > > Agree, wrapping is hard, but you can always use **kwargs. > > The pattern is sometimes useful for decorators (as Ryan Gonzalez > shows in this thread). But basically, you need this when 'None' > is a legitimate value for some optional argument, and you want > users to pass explicitly. > > For instance, see inspect.Signature.replace method: > http://hg.python.org/cpython/file/e4ee0b15cc4f/Lib/inspect.py#l2193 > > I'm not saying it's a very common thing to do, but sometimes > you need this, and having the need of doing some ad hoc hacks > is slightly annoying. ...but it would be yet another thing to learn. Plus, being it uncommon you would easily forget what's the "right way to do it" and likely come up with the usual object() hack anyway. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 17 00:32:25 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 16 Apr 2014 15:32:25 -0700 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: <534E5AAF.6080808@trueblade.com> Message-ID: <9BB6B12D-F1ED-4C47-935B-E2CB32A60D0B@yahoo.com> On Apr 16, 2014, at 4:50, Nick Coghlan wrote: > Note that my main interest here is in making commands like: > > py -3 -m pip install foo > > cross platform rather than Windows specific. At the moment we don't have a way to explicitly invoke Python 3 that works everywhere. > It might be a lot simpler to get just that than the full py script: If you just run python (off the path), unless -3 is specified, in which case you run python3 instead. Doesn't that over all known distros, third-party installations, and pristine local installs of 3.x? If you also want to make -2 portable, it seems like that requires some configuration (as long as some distros install 3.x as python, and others install 2.x as just python and not also python2). But is that a requirement! -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Thu Apr 17 08:30:17 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 17 Apr 2014 15:30:17 +0900 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: <9BB6B12D-F1ED-4C47-935B-E2CB32A60D0B@yahoo.com> References: <534E5AAF.6080808@trueblade.com> <9BB6B12D-F1ED-4C47-935B-E2CB32A60D0B@yahoo.com> Message-ID: <87eh0w4gee.fsf@uwakimon.sk.tsukuba.ac.jp> Andrew Barnert writes: > If you just run python (off the path), unless?-3 is specified, in > which case you run python3 instead. Doesn't that over all known > distros, third-party installations, and pristine local installs of > 3.x? I don't know if you consider MacPorts a "distro", but no, it doesn't install a /opt/local/bin/python3 link. You need to specify the minor version. From mertz at gnosis.cx Thu Apr 17 17:49:45 2014 From: mertz at gnosis.cx (David Mertz) Date: Thu, 17 Apr 2014 08:49:45 -0700 Subject: [Python-ideas] Add a datatype collections.ListView Message-ID: Following Brandon Rhodes very nice PyCon 2014 talk on datatypes, I was struck by increased guilt over a program I am developing. I spoke with him in the hallway about a good approach to improving performance while keeping code nice looking. The basic idea in my program is that I have a large(ish) list of tokens that I generate in parsing a special language, and I frequently take slices from this list--and often enough, slices out of those slices. In some cases, the most straightforward approach in code is a slice-to-end, e.g.: start = find_thing(tokens) construct_in = tokens[start:] Obviously, this winds up doing a lot of copying (of object references, not of actual data, but still). Often the above step is followed by something like: end = find_end(construct_in) construct = construct_in[:end] This second step isn't bad in my case since the actual construct will be dozens of tokens, not thousands or millions, and once I find it I want to keep it around and process it further. I realize, of course, that I could program my 'find_end()' differently so that it took a signature more like 'find_end(tokens, start=start)'. But with recursion and some other things I do, this becomes inelegant. What I'd really like is a "ListView" that acts something like NumPy's non-copying slices. However numpy, of course, only deals with arrays and matrices of uniform numeric types. I want a non-copying "slice" of a list of generic Python objects. Moreover, I believe that such a type is useful enough to be worth including in the collections module generally. As an initial implementation, I created the below. In the module self-tests, the performance increase is about 100x, but the particular ad-hoc benchmark I wrote isn't necessarily well-representative of all use-cases. % python3 ListView.py A bunch of slices from list: 1.29 seconds A bunch of slices from DummyListView: 1.19 seconds A bunch of slices from ListView: 0.01 seconds ------------------------------------------------------------------- ### ListView.py ### import sys from time import time from random import randint, random from collections import Sequence class DummyListView(Sequence): def __init__(self, l): self.list = l def __len__(self): return len(self.list) def __getitem__(self, i): return self.list[i] class ListView(Sequence): def __init__(self, seq, start=0, stop=None): if hasattr(seq, '__getitem__'): self.list = seq else: self.list = list(seq) self.start = start self.stop = len(self.list) if stop is None else stop def __len__(self): return self.stop - self.start def __getitem__(self, i): if isinstance(i, slice): start = self.start if i.start is None else self.start+i.start if i.stop is None: stop = self.stop else: stop = self.start + i.stop return ListView(self.list, start, stop) else: val = self.list[i+self.start] if i < 0: return val elif not self.start <= i+self.start < self.stop: raise IndexError("View of sequence [%d:%d], index %d" % ( self.start, self.stop, i)) return val def __str__(self): return "ListView of %d item list, slice [%d:%d]" % ( len(self.list), self.start, self.stop) def __repr__(self): return "ListView(%s)" % self.list[self.start:self.stop] def to_list(self): return list(self.list[self.start:self.stop]) class Thing(object): def __init__(self, x): self.x = x def __repr__(self): return "Thing(%f)" % self.x if __name__ == '__main__': NUM = 100000 things = [Thing(random()) for _ in range(NUM)] slices = [sorted((randint(0, NUM-1), randint(0, NUM-1))) for _ in range(100)] offset = randint(0, 100) for name, cls in (("list", list), ("DummyListView", DummyListView), ("ListView",ListView)): begin = time() s = "A bunch of slices from %s: " % name print(s.rjust(38), end='') sys.stdout.flush() l = cls(things) for i in range(8): for start, stop in slices: sl = l[start:stop] size = stop-start for i in range(3): subslice1 = sl[:offset] subslice2 = sl[offset:] print("%0.2f seconds" % (time()-begin)) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 17 18:28:45 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 Apr 2014 17:28:45 +0100 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: On Thu, Apr 17, 2014 at 4:49 PM, David Mertz wrote: > What I'd really like is a "ListView" that acts something like NumPy's > non-copying slices. However numpy, of course, only deals with arrays and > matrices of uniform numeric types. I want a non-copying "slice" of a list > of generic Python objects. FWIW, numpy arrays do indeed have to be of uniform type, but one of the supported uniform types is "arbitrary Python object". In [1]: a = np.array([1, "hello", None]) In [2]: a Out[2]: array([1, 'hello', None], dtype=object) In [3]: a[1:] Out[3]: array(['hello', None], dtype=object) There are other trade-offs though, e.g., no .append() or .find() methods. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From nad at acm.org Thu Apr 17 20:03:23 2014 From: nad at acm.org (Ned Deily) Date: Thu, 17 Apr 2014 11:03:23 -0700 Subject: [Python-ideas] py launcher for Linux? References: <534E5AAF.6080808@trueblade.com> <9BB6B12D-F1ED-4C47-935B-E2CB32A60D0B@yahoo.com> <87eh0w4gee.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: In article <87eh0w4gee.fsf at uwakimon.sk.tsukuba.ac.jp>, "Stephen J. Turnbull" wrote: > I don't know if you consider MacPorts a "distro", but no, it doesn't > install a /opt/local/bin/python3 link. You need to specify the minor > version. https://trac.macports.org/ticket/42747 -- Ned Deily, nad at acm.org From tjreedy at udel.edu Thu Apr 17 20:12:07 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Apr 2014 14:12:07 -0400 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: On 4/17/2014 11:49 AM, David Mertz wrote: > What I'd really like is a "ListView" that acts something like NumPy's > non-copying slices. Consider a more generic SeqView, especially if you do not intend to mutate through the view. I also suggest that you use a range object instead of .start, .stop, (and .step). Range gives you a lot for free, including slicing. Untested: class SeqView: def __init__(self, seq, rng) self.seq = seq self.rng = rng def __getitem__(self, index): if isinstance(index, slice): return Seqview(self.seq, self.rng[index]) else: return self.seq[self.rng[index]] # see below def __iter__(self): seq, rng = self.seq, self.rng for i in rng: yield seq[i] ... >>> r = range(3, 50, 3) >>> r[4] 15 >>> r[500] Traceback (most recent call last): File "", line 1, in r[500] IndexError: range object index out of range # You would want to catch this and change 'range' to SeqView >>> r[1:3] range(6, 12, 3) >>> r[slice(1,3)] range(6, 12, 3) >>> r[slice(1,3,2)] range(6, 12, 6) -- Terry Jan Reedy From ryan at ryanhiebert.com Thu Apr 17 20:28:19 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Thu, 17 Apr 2014 13:28:19 -0500 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: On Thu, Apr 17, 2014 at 1:12 PM, Terry Reedy wrote: > Range gives you a lot for free, including slicing. > I have wondered why Python has both slice and range, given the power that range has. Is there anything that slice can do that range can't, or are there some functional differences? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lemiant at hotmail.com Thu Apr 17 20:52:00 2014 From: lemiant at hotmail.com (Alex Rodrigues) Date: Thu, 17 Apr 2014 12:52:00 -0600 Subject: [Python-ideas] str.find() and friends support a lists of inputs Message-ID: It's a fairly common problem to want to .find() or .replace() or .split() any one of multiple characters. Currently the go to solution are: A: ? ?clean_number = formatted_phone_number.replace('-', '').replace('(', '').replace(')','').replace(' ','') B: ? ?get_rid_of = ["-","(",")"," "] ? ?clean_number = formatted_phone_number ? ?for ch in get_rid_of: ? ? ? ?clean_number = clean_number.replace(ch,'') C: ? ?import re ? ?clean_number = re.sub('[-\(\) ]', '', formatted_phone_number) While none of these is especially terrible they're also far from nice or clean. And whenever I'm faced with this kind of problem my automatic reaction is to type: clean_number = formatted_phone_number.replace(["-","(",")"," "],"") That is what I intuitively want to do, and it is the syntax other people often use when trying to describe that they want to replace multiple characters. I think this is because its semantics follow very logically from the original replace statement you learn. Instead of saying "replace this with that" it's saying "replace these with that". In the case of split() it gets even worse to try and split on multiple delimiters and you almost have to resort to using re. However for such simple cases re is serious overkill. You have to teach people about an entire new module, explain what regular expressions are and explain what new syntax like "(abc|def)" and "[abcdef]" means. When you could just use the string functions and list syntax you already understand. While re is an absolute life saver in certain situations, it is very non-performant for little one-of operations because it still has to compile a whole regular expression. Below is a quick test in iPython, intentionally bypassing the cache: In [1]: a = "a"*100+"b" In [2]: %timeit -n 1 -r 1 a.find('b') 1 loops, best of 1: 3.31 ?s per loop In [3]: import re In [4]: %%timeit -n 1 -r 1 re.purge() ? ?...: re.search('[b]', 'a') ? ?...: 1 loops, best of 1: 132 ?s per loop So for all those reasons, this is what I propose. Making .find() support lists of targets, .split() support lists of delimiters and .replace() support lists of targets. The objective of this is not to support all possible permutations of string operations, I expect there are many cases that this will not solve, however it is meant to make the built in string operations support a slightly larger set of very common operations which fit intuitively with the existing syntax. I'd also like to note what my own concerns were with this idea: My first concern was that this might break existing code. But a quick check shows that this is invalid syntax at the moment, so it doesn't affect backwards compatibility at all. My second concern was with handling the possibility of collisions within the list (i.e. "snowflake".replace(['snow', 'snowflake'])) This could be ameliorated by explicitly deciding that whichever match begins earlier will be applied before considering the others and if they start at the same position the one earlier in the list will be resolved first. However, I'd argue that if you really need explicit control over the match order of words which contain each other that's a pretty good time to start considering regular expressions. Below are a sampling of questions from Stack Overflow which would have benefited from the existence of this syntax. http://stackoverflow.com/questions/21859203/how-to-replace-multiple-characters-in-a-string http://stackoverflow.com/questions/4998629/python-split-string-with-multiple-delimiters http://stackoverflow.com/questions/10017147/python-replace-characters-in-string http://stackoverflow.com/questions/14215338/python-remove-multiple-character-in-list-of-string Cheers, - Alex From 7vsfeu4pxg at snkmail.com Thu Apr 17 21:14:11 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Thu, 17 Apr 2014 19:14:11 +0000 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal Message-ID: <22741-1397762051-908018@sneakemail.com> Hello everybody. I firstly proposed this syntax in the python-list. As I already wrote there, I read PEP 3103 and I'm not completely satisfied by any of the proposed solutions. My first idea was a little different, this is my final proposal after a short but good brainstorm: switch_stmt ::= "switch" switch_expr "case" case_expr ":" suite ("case" | "elcase" case_expr ":" suite)* ["else" ":" suite] switch_expr ::= expression case_expr ::= expression_list - if case_expr is a tuple, the case suite will be executed if switch_expr is a member of the tuple - if case_expr is not a tuple, the case suite will be executed if switch_expr == case_expr - if a case_expr is checked, any subsequent elcase statements are skipped, and the next case statement is performed, of there's one. This is completely identical to if - elif. Example: briefing_days = ("Tue", "Thu") normal_days = ("Mon", "Wed", "Fri") switch day case normal_days + briefing_days: go_to_work = True day_type = "weekday" case normal_days: lunch_time = datetime.time(12) meeting_time = datetime.time(14) elcase briefing_days: lunch_time = datetime.time(11, 30) meeting_time = datetime.time(12, 30) else: go_to_work = False day_type = "festive" lunch_time = None meeting_time =None A simpler example: switch tarot case 0: card = "Fool" elcase 1: card = "Alan Moore" elcase 2: card = "High Priestess" Some remarks: 1. switch is on the same line of the first case. This is because alternatives in PEP 3103 seems unpythonic to me 2. I decided to not use already existing keywords like "if" or "in", since this will be misleading 3. I preferred case / elcase instead of using break or continue keyword because: a. break / continue can confuse, since they are used in loop statements, and this is not a loop statement, as Ethan Furman pointed out in the other mailing list b. break / continue is useful only if it can be used not only as final command, so you can skip the rest of the current case suite. If it must be the final command, it's the same of case - elcase. IMHO there's no much real need to allow this c. case - elcase syntax is much more readable and less confusing of a break / continue d. case - elcase syntax is identical to if - elif, so people will be more familiar with it 4. notice that you can put a "case" statement after an "elcase" one. For example: switch cpu case "A6-6400K", "A8-6600K" clock = 3.9 elcase "A10-6790K", "A10-6800B" clock = 4.1 case "A6-6400K" tdp = 65 elcase "A8-6600K", "A10-6790K", "A10-6800B" tdp = 100 is equivalent to if cpu in ("A6-6400K", "A8-6600K") clock = 3.9 elif cpu in ("A10-6790K", "A10-6800B") clock = 4.1 if cpu == "A6-6400K" tdp = 65 elif cpu in ("A8-6600K", "A10-6790K", "A10-6800B") tdp = 100 From stephane at wirtel.be Thu Apr 17 21:28:05 2014 From: stephane at wirtel.be (=?utf-8?q?St=C3=A9phane?= Wirtel) Date: Thu, 17 Apr 2014 15:28:05 -0400 Subject: [Python-ideas] A library for the deprecation of a function/class Message-ID: With the CPython sprint, I was thinking about a lib to mark a function/class as deprecated. Example: #!/usr/bin/env python import sys import deprecation # from an external lib or from a stdlib module __version__ = (1,0) @deprecation.deprecated(python=(3,7,), msg='use foo_v2') def foo(): pass @deprecation.deprecated(program=(1,2), msg='use bar_v2') def bar(): pass @deprecation.deprecated(python=(3,7), msg='use inspect.signature()') @deprecation.deprecated(python=(3,7), to_use=inspect.signature) def getfullargspec(*args, **kwargs): pass The deprecated decorator should check the version of the software and the version of Python if asked with the arguments. it will raise warnings.warn with PendingDeprecationWarning or DeprecationWarning. Can be used in the documentation, via introspection. It's just an idea, there is no code, no specs but if you are interested I think I can try to propose a real solution with an external library and if this idea seems to be interesting, I will propose a PEP for 3.5 or 3.6. The interest of this lib, we can parse the source code and find the deprecated functions/classes with a small tool and the maintenance of the code should be improved. Please, could you give me your feedback? Thank you, Stephane -- St?phane Wirtel - http://wirtel.be - @matrixise From guido at python.org Thu Apr 17 21:31:31 2014 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Apr 2014 12:31:31 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <22741-1397762051-908018@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> Message-ID: I don't want to discourage you too much, but I think that adding a switch statement comes *very* low on the list of improvements we would like to make in Python 3.5. We should probably focus on speed (or aspects of it, like startup), language features that help porting Python 2 code to it (e.g. bytes formatting), and things that improve the user experience of getting started with Python on a new machine (e.g. pip/venv). Or perhaps stdlib issues like an asyncio-infused variation of WSGI. I've probably missed a few focus areas, but I still very much doubt we'll be adding a switch statement -- it's a "sexy" language design issue (like anonymous functions) but that's not what will help Python compete. On Thu, Apr 17, 2014 at 12:14 PM, Lucas Malor <7vsfeu4pxg at snkmail.com>wrote: > Hello everybody. I firstly proposed this syntax in the python-list. As I > already wrote there, I read PEP 3103 and I'm not completely satisfied by > any of the proposed solutions. > > My first idea was a little different, this is my final proposal after a > short but good brainstorm: > > switch_stmt ::= "switch" switch_expr "case" case_expr ":" suite > ("case" | "elcase" case_expr ":" suite)* > ["else" ":" suite] > switch_expr ::= expression > case_expr ::= expression_list > > - if case_expr is a tuple, the case suite will be executed if switch_expr > is a member of the tuple > - if case_expr is not a tuple, the case suite will be executed if > switch_expr == case_expr > - if a case_expr is checked, any subsequent elcase statements are > skipped, and the next case statement is performed, of there's one. This is > completely identical to if - elif. > > Example: > > briefing_days = ("Tue", "Thu") > normal_days = ("Mon", "Wed", "Fri") > > switch day case normal_days + briefing_days: > go_to_work = True > day_type = "weekday" > case normal_days: > lunch_time = datetime.time(12) > meeting_time = datetime.time(14) > elcase briefing_days: > lunch_time = datetime.time(11, 30) > meeting_time = datetime.time(12, 30) > else: > go_to_work = False > day_type = "festive" > lunch_time = None > meeting_time =None > > A simpler example: > > switch tarot case 0: > card = "Fool" > elcase 1: > card = "Alan Moore" > elcase 2: > card = "High Priestess" > > > Some remarks: > > 1. switch is on the same line of the first case. This is because > alternatives in PEP 3103 seems unpythonic to me > 2. I decided to not use already existing keywords like "if" or "in", since > this will be misleading > 3. I preferred case / elcase instead of using break or continue keyword > because: > a. break / continue can confuse, since they are used in loop > statements, and this is not a loop statement, as Ethan Furman pointed out > in the other mailing list > b. break / continue is useful only if it can be used not only as final > command, so you can skip the rest of the current case suite. If it must be > the final command, it's the same of case - elcase. IMHO there's no much > real need to allow this > c. case - elcase syntax is much more readable and less confusing of a > break / continue > d. case - elcase syntax is identical to if - elif, so people will be > more familiar with it > 4. notice that you can put a "case" statement after an "elcase" one. For > example: > > switch cpu case "A6-6400K", "A8-6600K" > clock = 3.9 > elcase "A10-6790K", "A10-6800B" > clock = 4.1 > case "A6-6400K" > tdp = 65 > elcase "A8-6600K", "A10-6790K", "A10-6800B" > tdp = 100 > > is equivalent to > > if cpu in ("A6-6400K", "A8-6600K") > clock = 3.9 > elif cpu in ("A10-6790K", "A10-6800B") > clock = 4.1 > > if cpu == "A6-6400K" > tdp = 65 > elif cpu in ("A8-6600K", "A10-6790K", "A10-6800B") > tdp = 100 > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 17 22:14:23 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 17 Apr 2014 13:14:23 -0700 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: On Apr 17, 2014, at 11:52, Alex Rodrigues wrote: > It's a fairly common problem to want to .find() or .replace() or .split() any one of multiple characters. I like your solution, except for one thing. Explicitly requiring a list of arguments rather than, say, a tuple or an iterator, seems unnecessarily restrictive. However, allowing any iterable of strings doesn't work because a string is itself an iterable of strings. There are a few cases where Python deals with this problem by treating tuples specially (e.g., % formatting), but I don't think anyone wants to extend that solution. You could almost handle these problems with *args (replace replaces any of args[:-1] with args[-1]), except that all of them have optional parameters at the end. You could have a keyword-only argument to specify an iterable of strings (in which case you can't use any positional arguments), but that's a pretty weird interface. Or you could just add some new methods: split_any, replace_any, etc. But str already has a lot of methods; do we really want more? I'd love to see an answer that works here, because I agree that it would make a lot of code simpler, and especially code that novices want to write. From ryan at ryanhiebert.com Thu Apr 17 22:28:36 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Thu, 17 Apr 2014 15:28:36 -0500 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: I like the idea. I agree with the assertion that has been discussed previously that a string really shouldn't be iterable. Because of that, I think that explicitly checking if it is str, and if not, using it as an iterator, would be appropriate. On Thu, Apr 17, 2014 at 3:14 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Apr 17, 2014, at 11:52, Alex Rodrigues wrote: > > > It's a fairly common problem to want to .find() or .replace() or > .split() any one of multiple characters. > > I like your solution, except for one thing. Explicitly requiring a list of > arguments rather than, say, a tuple or an iterator, seems unnecessarily > restrictive. However, allowing any iterable of strings doesn't work because > a string is itself an iterable of strings. > > There are a few cases where Python deals with this problem by treating > tuples specially (e.g., % formatting), but I don't think anyone wants to > extend that solution. > > You could almost handle these problems with *args (replace replaces any of > args[:-1] with args[-1]), except that all of them have optional parameters > at the end. > > You could have a keyword-only argument to specify an iterable of strings > (in which case you can't use any positional arguments), but that's a pretty > weird interface. > > Or you could just add some new methods: split_any, replace_any, etc. But > str already has a lot of methods; do we really want more? > > I'd love to see an answer that works here, because I agree that it would > make a lot of code simpler, and especially code that novices want to write. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Apr 17 23:11:44 2014 From: mertz at gnosis.cx (David Mertz) Date: Thu, 17 Apr 2014 14:11:44 -0700 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: On Thu, Apr 17, 2014 at 9:28 AM, Nathaniel Smith wrote: >FWIW, numpy arrays do indeed have to be of uniform type, but one of >the supported uniform types is "arbitrary Python object". I did not know that! However, now that I do: % python3 ListView.py A bunch of slices from list: 3.76 seconds A bunch of slices from DummyListView: 3.33 seconds A bunch of slices from ListView: 0.03 seconds A bunch of slices from numpy.array: 0.30 seconds The only things I changed are to add 'import numpy' at the top, and add 'np.array' to the loop of containers to try out (I also bumped up the iterations of the two loops from 8/3 times to 10/5 times just to make things take a little longer and get slightly more accurate timing I hope). On Thu, Apr 17, 2014 at 9:28 AM, Nathaniel Smith wrote: > On Thu, Apr 17, 2014 at 4:49 PM, David Mertz wrote: > > What I'd really like is a "ListView" that acts something like NumPy's > > non-copying slices. However numpy, of course, only deals with arrays and > > matrices of uniform numeric types. I want a non-copying "slice" of a > list > > of generic Python objects. > > FWIW, numpy arrays do indeed have to be of uniform type, but one of > the supported uniform types is "arbitrary Python object". > > In [1]: a = np.array([1, "hello", None]) > > In [2]: a > Out[2]: array([1, 'hello', None], dtype=object) > > In [3]: a[1:] > Out[3]: array(['hello', None], dtype=object) > > There are other trade-offs though, e.g., no .append() or .find() methods. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Apr 17 23:22:25 2014 From: mertz at gnosis.cx (David Mertz) Date: Thu, 17 Apr 2014 14:22:25 -0700 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: Using range objects might be worthwhile. However, your example code, Terry, won't behave well if you initialize with a generic iterable. You need the check for 'hasattr(seq, "__getitem__")' or something equivalent in the initializer. But if you have that, you need to make a decision of *what* concrete sequence to instantiate the iterable as, with list() being the obvious choice. E.g. I can do: lv = ListView(Thing(random()) for _ in range(100000)) But you can't do: sv = SeqView(Thing(random()) for _ in range(100000)) I think, however, there's no reason at all why I should have my .to_list() method. That's accomplished more naturally within a special method just as: concrete = list(lv[5:10]) On Thu, Apr 17, 2014 at 11:12 AM, Terry Reedy wrote: > On 4/17/2014 11:49 AM, David Mertz wrote: > > What I'd really like is a "ListView" that acts something like NumPy's >> non-copying slices. >> > > Consider a more generic SeqView, especially if you do not intend to mutate > through the view. I also suggest that you use a range object instead of > .start, .stop, (and .step). Range gives you a lot for free, including > slicing. Untested: > > class SeqView: > def __init__(self, seq, rng) > self.seq = seq > self.rng = rng > > def __getitem__(self, index): > if isinstance(index, slice): > return Seqview(self.seq, self.rng[index]) > else: > return self.seq[self.rng[index]] # see below > def __iter__(self): > seq, rng = self.seq, self.rng > for i in rng: > yield seq[i] > ... > > >>> r = range(3, 50, 3) > >>> r[4] > 15 > > >>> r[500] > Traceback (most recent call last): > File "", line 1, in > r[500] > IndexError: range object index out of range > # You would want to catch this and change 'range' to SeqView > > >>> r[1:3] > range(6, 12, 3) > >>> r[slice(1,3)] > range(6, 12, 3) > >>> r[slice(1,3,2)] > range(6, 12, 6) > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 17 23:41:56 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 17 Apr 2014 14:41:56 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <22741-1397762051-908018@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> Message-ID: <1F992A6C-6C28-4447-A637-487838D7547E@yahoo.com> On Apr 17, 2014, at 12:14, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > switch_stmt ::= "switch" switch_expr "case" case_expr ":" suite > ("case" | "elcase" case_expr ":" suite)* > ["else" ":" suite] > switch_expr ::= expression > case_expr ::= expression_list Any reason to use the keywords that only exist in C-family languages when the functionality isn't actually like C? > - if case_expr is a tuple, the case suite will be executed if switch_expr is a member of the tuple So there's no way to switch on a tuple? > - if case_expr is not a tuple, the case suite will be executed if switch_expr == case_expr > - if a case_expr is checked, any subsequent elcase statements are skipped, and the next case statement is performed, of there's one. This is completely identical to if - elif. While you can figure out what this means with a bit of work, it seems to add a lot of conceptual complexity, and I can't think of any other language that does this. Meanwhile, if you're trying to design a better and more powerful switch statement than other languages have, why not consider features that other languages do have, like pattern matching cases? Maybe it would be too hard to implement, or would conflict with another part of the feature, or just wouldn't fit in with Python, but if so, a switch proposal should try to explain that. From abarnert at yahoo.com Thu Apr 17 23:49:32 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 17 Apr 2014 14:49:32 -0700 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: <0540BF63-1A71-41C2-A38C-8B9427196AC9@yahoo.com> On Apr 17, 2014, at 13:28, Ryan Hiebert wrote: > I like the idea. I agree with the assertion that has been discussed previously that a string really shouldn't be iterable. I don't want to reopen that whole argument here, except to say that there are strong cases for both sides and nobody's managed to convince everyone to their side. At any rate, even if you don't think a string should be an iterable, it definitely _is_ one in the language as it is today. > Because of that, I think that explicitly checking if it is str, and if not, using it as an iterator, would be appropriate. You mean as an iterable, not as an iterator, right? But anyway, that would be very odd. Is there anywhere else in the stdlib that strings are treated specially and not iterated over? (As I mentioned before, there are cases where tuples are treated specially, but I don't think anyone considers that a good thing, or wants to take it any further.) Also, you have to be careful about what you mean by "string". Does that mean str, str or any subclass, anything that quacks like a string in a certain context, ...? And then, what's the parallel definition for bytes and bytearray methods? (Maybe there should be ABCs for String, MutableString, ByteString, and MutableByteString to provide a better answer to those questions?) > On Thu, Apr 17, 2014 at 3:14 PM, Andrew Barnert wrote: >> On Apr 17, 2014, at 11:52, Alex Rodrigues wrote: >> >> > It's a fairly common problem to want to .find() or .replace() or .split() any one of multiple characters. >> >> I like your solution, except for one thing. Explicitly requiring a list of arguments rather than, say, a tuple or an iterator, seems unnecessarily restrictive. However, allowing any iterable of strings doesn't work because a string is itself an iterable of strings. >> >> There are a few cases where Python deals with this problem by treating tuples specially (e.g., % formatting), but I don't think anyone wants to extend that solution. >> >> You could almost handle these problems with *args (replace replaces any of args[:-1] with args[-1]), except that all of them have optional parameters at the end. >> >> You could have a keyword-only argument to specify an iterable of strings (in which case you can't use any positional arguments), but that's a pretty weird interface. >> >> Or you could just add some new methods: split_any, replace_any, etc. But str already has a lot of methods; do we really want more? >> >> I'd love to see an answer that works here, because I agree that it would make a lot of code simpler, and especially code that novices want to write. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Thu Apr 17 23:53:43 2014 From: phd at phdru.name (Oleg Broytman) Date: Thu, 17 Apr 2014 23:53:43 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <1F992A6C-6C28-4447-A637-487838D7547E@yahoo.com> References: <22741-1397762051-908018@sneakemail.com> <1F992A6C-6C28-4447-A637-487838D7547E@yahoo.com> Message-ID: <20140417215343.GA18833@phdru.name> While I don't think it's a good addition for Python... On Thu, Apr 17, 2014 at 02:41:56PM -0700, Andrew Barnert wrote: > On Apr 17, 2014, at 12:14, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > > > switch_stmt ::= "switch" switch_expr "case" case_expr ":" suite > > ("case" | "elcase" case_expr ":" suite)* > > ["else" ":" suite] > > switch_expr ::= expression > > case_expr ::= expression_list > > > - if case_expr is a tuple, the case suite will be executed if switch_expr is a member of the tuple > > So there's no way to switch on a tuple? A tuple can be a member of a bigger tuple: >>> (1, 'b') in ((1, 'a'), (1, 'b'), (1, 'a', 'b')) True Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ryan at ryanhiebert.com Fri Apr 18 00:21:07 2014 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Thu, 17 Apr 2014 17:21:07 -0500 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: <0540BF63-1A71-41C2-A38C-8B9427196AC9@yahoo.com> References: <0540BF63-1A71-41C2-A38C-8B9427196AC9@yahoo.com> Message-ID: On Thu, Apr 17, 2014 at 4:49 PM, Andrew Barnert wrote: > On Apr 17, 2014, at 13:28, Ryan Hiebert wrote: > > Because of that, I think that explicitly checking if it is str, and if > not, using it as an iterator, would be appropriate. > > You mean as an iterable, not as an iterator, right? > Yes. > > But anyway, that would be very odd. Is there anywhere else in the stdlib > that strings are treated specially and not iterated over? (As I mentioned > before, there are cases where tuples are treated specially, but I don't > think anyone considers that a good thing, or wants to take it any further.) > > Also, you have to be careful about what you mean by "string". Does that > mean str, str or any subclass, anything that quacks like a string in a > certain context, ...? And then, what's the parallel definition for bytes > and bytearray methods? (Maybe there should be ABCs for String, > MutableString, ByteString, and MutableByteString to provide a better answer > to those questions?) > How about defining it as "anything that currently works in those contexts"? Currently, it allows str and subclasses, but not bytes, because they cannot be converted implicitly to str. So, try it how it works now, and if that fails, try using it as an iterable. Are there similar methods on bytes which might make that definition confusing? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Apr 18 01:05:05 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 18 Apr 2014 09:05:05 +1000 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: On Fri, Apr 18, 2014 at 4:52 AM, Alex Rodrigues wrote: > Below is a quick test in iPython, intentionally bypassing the cache: > > In [1]: a = "a"*100+"b" > > In [2]: %timeit -n 1 -r 1 a.find('b') > 1 loops, best of 1: 3.31 ?s per loop > > In [3]: import re > > In [4]: %%timeit -n 1 -r 1 re.purge() > ...: re.search('[b]', 'a') > ...: > 1 loops, best of 1: 132 ?s per loop > I'm always dubious of micro-benchmarks, especially when caches have to be deliberately bypassed. How does the time compare if you *don't* purge the cache? After all, compiling an RE once and using it lots of times is exactly how they're meant to be used. Yes, it would be potentially cleaner to offer a list of strings to .find(); but maybe reaching for a regex is the right thing to do. Last night I wanted to rename a whole bunch of files thus: "DoYouWannaBuildASnowman.mkv" -> "Frozen - Do You Wanna Build A Snowman.mkv". Constant text at the beginning, then add a space before every capital letter. Heretical or not, I went regex. :) ChrisA From lemiant at hotmail.com Fri Apr 18 01:14:49 2014 From: lemiant at hotmail.com (Alex Rodrigues) Date: Thu, 17 Apr 2014 17:14:49 -0600 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: , , , <0540BF63-1A71-41C2-A38C-8B9427196AC9@yahoo.com>, Message-ID: Currently .endswith() and .startswith() accept a str, unicode, or tuple and use the tuple exactly in the same way this would. That might not be a bad place to start when thinking about which types to support. Date: Thu, 17 Apr 2014 17:21:07 -0500 Subject: Re: [Python-ideas] str.find() and friends support a lists of inputs From: ryan at ryanhiebert.com To: abarnert at yahoo.com CC: abarnert at yahoo.com.dmarc.invalid; lemiant at hotmail.com; python-ideas at python.org On Thu, Apr 17, 2014 at 4:49 PM, Andrew Barnert wrote: On Apr 17, 2014, at 13:28, Ryan Hiebert wrote: Because of that, I think that explicitly checking if it is str, and if not, using it as an iterator, would be appropriate. You mean as an iterable, not as an iterator, right? Yes. But anyway, that would be very odd. Is there anywhere else in the stdlib that strings are treated specially and not iterated over? (As I mentioned before, there are cases where tuples are treated specially, but I don't think anyone considers that a good thing, or wants to take it any further.) Also, you have to be careful about what you mean by "string". Does that mean str, str or any subclass, anything that quacks like a string in a certain context, ...? And then, what's the parallel definition for bytes and bytearray methods? (Maybe there should be ABCs for String, MutableString, ByteString, and MutableByteString to provide a better answer to those questions?) How about defining it as "anything that currently works in those contexts"? Currently, it allows str and subclasses, but not bytes, because they cannot be converted implicitly to str. So, try it how it works now, and if that fails, try using it as an iterable. Are there similar methods on bytes which might make that definition confusing? -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Apr 18 01:16:28 2014 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 18 Apr 2014 00:16:28 +0100 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: <535060CC.5010509@mrabarnett.plus.com> On 2014-04-17 21:14, Andrew Barnert wrote: > On Apr 17, 2014, at 11:52, Alex Rodrigues > wrote: > >> It's a fairly common problem to want to .find() or .replace() or >> .split() any one of multiple characters. > > I like your solution, except for one thing. Explicitly requiring a > list of arguments rather than, say, a tuple or an iterator, seems > unnecessarily restrictive. However, allowing any iterable of strings > doesn't work because a string is itself an iterable of strings. > > There are a few cases where Python deals with this problem by > treating tuples specially (e.g., % formatting), but I don't think > anyone wants to extend that solution. > str.startswith and str.endswith accept either a string or a tuple of strings, but not a list of strings nor any other iterable, so does it matter if str.find, etc, accepted a tuple but not a list? > You could almost handle these problems with *args (replace replaces > any of args[:-1] with args[-1]), except that all of them have > optional parameters at the end. > > You could have a keyword-only argument to specify an iterable of > strings (in which case you can't use any positional arguments), but > that's a pretty weird interface. > > Or you could just add some new methods: split_any, replace_any, etc. > But str already has a lot of methods; do we really want more? > > I'd love to see an answer that works here, because I agree that it > would make a lot of code simpler, and especially code that novices > want to write. From tjreedy at udel.edu Fri Apr 18 02:21:24 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Apr 2014 20:21:24 -0400 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: On 4/17/2014 5:22 PM, David Mertz wrote: > Using range objects might be worthwhile. However, your example code, > Terry, won't behave well if you initialize with a generic iterable. That is why I said 'sequence', not 'iterable'. If someone ignores the signature of a class/function, that is his problem. Anyway, my 'example code' was obviously an illustrative but incomplete outline, with missing details like docstrings, type checks, and exception catching left for you to fill in. > You need the check for 'hasattr(seq, "__getitem__")' or something equivalent > in the initializer. But if you have that, you need to make a decision > of *what* concrete sequence to instantiate the iterable as, with list() > being the obvious choice. Your presented use case is that you already have a long (concrete) sequence and want to take views instead of slices to save memory. > E.g. I can do: > > lv = ListView(Thing(random()) for _ in range(100000)) > > But you can't do: > > sv = SeqView(Thing(random()) for _ in range(100000)) This does not make much sense and is outside your use case. -- Terry Jan Reedy From mertz at gnosis.cx Fri Apr 18 02:59:08 2014 From: mertz at gnosis.cx (David Mertz) Date: Thu, 17 Apr 2014 17:59:08 -0700 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: Sure, my example of initializing with an iterable isn't quite the case I need myself, but I was just thinking of what a general-purpose collection should do. And actually, if I had started out using this collection, I might have structured it as initializing with an iterable rather than collecting my objects with list.append(). On Thu, Apr 17, 2014 at 5:21 PM, Terry Reedy wrote: > On 4/17/2014 5:22 PM, David Mertz wrote: > >> Using range objects might be worthwhile. However, your example code, >> Terry, won't behave well if you initialize with a generic iterable. >> > > That is why I said 'sequence', not 'iterable'. If someone ignores the > signature of a class/function, that is his problem. > > Anyway, my 'example code' was obviously an illustrative but incomplete > outline, with missing details like docstrings, type checks, and exception > catching left for you to fill in. > > > You need the check for 'hasattr(seq, "__getitem__")' or something >> equivalent >> in the initializer. But if you have that, you need to make a decision >> of *what* concrete sequence to instantiate the iterable as, with list() >> being the obvious choice. >> > > Your presented use case is that you already have a long (concrete) > sequence and want to take views instead of slices to save memory. > > > E.g. I can do: >> >> lv = ListView(Thing(random()) for _ in range(100000)) >> >> But you can't do: >> >> sv = SeqView(Thing(random()) for _ in range(100000)) >> > > This does not make much sense and is outside your use case. > > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Apr 18 03:31:24 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Apr 2014 21:31:24 -0400 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: On 4/17/2014 2:52 PM, Alex Rodrigues wrote: > It's a fairly common problem to want to .find() or .replace() or .split() any one of multiple characters. > Currently the go to solution are: For replace, you left out the actual solution. >>> telnum = '(800), 555-1234' >>> telnum.translate(str.maketrans('','','(- ,)')) '8005551234' For finding any of multiple chars, a loop that does just what you want is easy. targets = '(- ,)' for i, c in enumerate(s): if c in targets: break else: References: Message-ID: <20140418013534.GD28400@ando> On Thu, Apr 17, 2014 at 01:28:19PM -0500, Ryan Hiebert wrote: > On Thu, Apr 17, 2014 at 1:12 PM, Terry Reedy wrote: > > > Range gives you a lot for free, including slicing. > > > > I have wondered why Python has both slice and range, given the power that > range has. Is there anything that slice can do that range can't, or are > there some functional differences? Because one is screwdriver and the other is a spanner? (Just don't ask me which one is which :-) But seriously, while they have similar signatures, they are semantically very different. range objects are concrete interators that produce numeric values, and all three arguments must be integers: py> range(2, 3, None) Traceback (most recent call last): File "", line 1, in TypeError: 'NoneType' object cannot be interpreted as an integer while slices are an extremely generic, type-agnostic mechanism for pointing to multiple indexes of some sequence: py> slice("spam", 2.5, 17) slice('spam', 2.5, 17) Slices have no meaning of their own, it's up to the sliced object to interpret them. The above slice will raise TypeError when used on a list, but may be accepted happily by some other custom sequence type. As far as built-in objects go, slices with negative indexes, or with end=None, can't be interpreted as indexes until you know which sequence they are applied to: range(1, 6, 2) always yields 1, 3, 5. slice(1, 6, 2) always refers to indexes 1, 3, 5. but range(1, -6, 2) is always empty. slice(1, -6, 2) may be empty, or it could refer to index 1 alone, or to indexes 1, 3, or indexes 1, 3, 5, or 1, 3, 5, 7, or ... In general, you cannot tell what indexes the slice will cover until you know what sequence it has been applied to. Built-in lists and tuples allow slices to extend past the ends of the sequence. That concept is meaningless with range. So despite the fact that slice() and range() have similar signatures, they are used differently, and have very different semantics. -- Steven From tjreedy at udel.edu Fri Apr 18 03:46:15 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Apr 2014 21:46:15 -0400 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: On 4/17/2014 4:28 PM, Ryan Hiebert wrote: > I like the idea. The idea depends on being able to iterate through strings. > I agree with the assertion that has been discussed > previously that a string really shouldn't be iterable. ...except when they should be. This assertion is off-topic for python-ideas. If you bring it up on python-list, I will say more. -- Terry Jan Reedy From steve at pearwood.info Fri Apr 18 03:49:32 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 Apr 2014 11:49:32 +1000 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: <20140418014932.GE28400@ando> On Thu, Apr 17, 2014 at 01:14:23PM -0700, Andrew Barnert wrote: > On Apr 17, 2014, at 11:52, Alex Rodrigues wrote: > > > It's a fairly common problem to want to .find() or .replace() or > > .split() any one of multiple characters. > > I like your solution, except for one thing. Explicitly requiring a > list of arguments rather than, say, a tuple or an iterator, seems > unnecessarily restrictive. However, allowing any iterable of strings > doesn't work because a string is itself an iterable of strings. > > There are a few cases where Python deals with this problem by treating > tuples specially (e.g., % formatting), but I don't think anyone wants > to extend that solution. I do! That makes the decision really simple: if the argument is a tuple, it is treated as multiple values, otherwise it is treated as a single value. That's how other string methods operate: py> 'abcd'.startswith(('xyz', 'abc')) True py> 'abcd'.startswith(['xyz', 'abc']) Traceback (most recent call last): File "", line 1, in TypeError: startswith first arg must be str or a tuple of str, not list so it's quite easy to learn, with no concerns about whether or not the argument will accept a set or a dict or iterators... -- Steven From steve at pearwood.info Fri Apr 18 04:22:22 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 Apr 2014 12:22:22 +1000 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: <20140418022222.GF28400@ando> On Thu, Apr 17, 2014 at 09:31:24PM -0400, Terry Reedy wrote: > On 4/17/2014 2:52 PM, Alex Rodrigues wrote: > >It's a fairly common problem to want to .find() or .replace() or .split() > >any one of multiple characters. > >Currently the go to solution are: > > For replace, you left out the actual solution. > > >>> telnum = '(800), 555-1234' > >>> telnum.translate(str.maketrans('','','(- ,)')) > '8005551234' That solution only works when you want to replace single characters. It doesn't help to replace generic substrings: "That's one small step for a man, one giant leap for mankind.".replace( 'man', 'person') Naively looping over your input may not work. for term in (a, b, c): s = s.replace(term, x) is not always the same as doing the replacements in a single pass. It seems like it ought to be the same, until you run into a situation like this: py> recipe = "Add one capsicum to the stew..." py> for term in ("capsicum", "chilli pepper", "pepper"): ... recipe = recipe.replace(term, "bell pepper") ... py> print(recipe) Add one bell bell pepper to the stew... Oops! If the search terms are not known until runtime, you may have a lot of difficulty doing the replacements in an order that doesn't cause problems like this. There are ways around this problem, but they're tricky to get right. Although this is easily done using a regex, it does require the user learn about regexes, which may be overkill. They have a steep learning curve and can be intimidating to beginners. In order of preference, I'd prefer: +1 allow str.find, .index and .replace to take tuple arguments to search for multiple substrings in a single pass; +0.5 to add helper functions in the string module findany replaceany. (No need for an indexany.) -0 tell the user to "just use a regex". (Perhaps we could give a couple of regex recipes in the Python FAQ or the re docs?) -- Steven From tjreedy at udel.edu Fri Apr 18 06:44:46 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Apr 2014 00:44:46 -0400 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: <20140418022222.GF28400@ando> References: <20140418022222.GF28400@ando> Message-ID: On 4/17/2014 10:22 PM, Steven D'Aprano wrote: > On Thu, Apr 17, 2014 at 09:31:24PM -0400, Terry Reedy wrote: >> On 4/17/2014 2:52 PM, Alex Rodrigues wrote: >>> It's a fairly common problem to want to .find() or .replace() or .split() >>> any one of multiple characters. >>> Currently the go to solution are: >> >> For replace, you left out the actual solution. >> >>>>> telnum = '(800), 555-1234' >>>>> telnum.translate(str.maketrans('','','(- ,)')) >> '8005551234' > > That solution only works when you want to replace single characters. That was the problem Alex presented and solved 3 ways other than the above. > It doesn't help to replace generic substrings: This is definite a harder problem. The easiest solution would be import re def replace_multiple_targets(string, target_tuple, replacement): pattern = '|'.join(target_tuple) return re.sub(pattern, replacement, string) print(replace_multiple_targets("Add one capsicum to the stew...", ("capsicum", "chilli pepper", "pepper"), "bell pepper") ) # Add one bell pepper to the stew... > Although this is easily done using a regex, it does require the user > learn about regexes, which may be overkill. They have a steep learning > curve and can be intimidating to beginners. On of the problems is the mismatch of apis. str.replace(string, pattern, repl[, count]) # versus re.sub(pattern, repl, string, count=0, flags=0) The function/method name is different, the parameter order is different (partly for good reason), and the count default is different (at least as presented). Ugh. I can never remember this and use help each time, at least for the re version. The other reason Alex does not like re is that it is 'slow'. In this case, the medium difficulty problem (see below), matching any of multiple strings, has medium speed solutions. https://en.wikipedia.org/wiki/Aho-Corasick_string_matching_algorithm https://en.wikipedia.org/wiki/Rabin-Karp_algorithm > In order of preference, I'd prefer: > > +1 allow str.find, .index and .replace to take tuple arguments to search > for multiple substrings in a single pass; + delta > +0.5 to add helper functions in the string module findany replaceany. > (No need for an indexany.) - something > -0 tell the user to "just use a regex". + epsilon > (Perhaps we could give a couple of regex recipes in the Python FAQ or > the re docs?) Like the above? My worry about the request is this. Python has many pairs of functions with one being a slow generic version and the other being a fast special case version. The fast and slow paths within the interpreter are one example. The str versus re pairs are another. There are often in-between, more generic but not fully, medium speed versions, perhaps more than one. How many of these should be provide? My worry is making the language overall harder to maintain and/or use by providing a multitude of in-between functions. In this case, the ease of simply allowing tuples as an alternative input, with a fairly obvious meaning, is a plus. An addition to your list is this. Add an new msm (multiple string match) module that would expose one (or both?) of the algorithms above in a match function and provide methods or functions like those in the str and re modules. I would be tempted to use the str versions of the apis as possible, and omit a compile function and the corresponding methods. (The hidden cache should be enough to avoid constanct recompiles.) -- Terry Jan Reedy From abarnert at yahoo.com Fri Apr 18 07:03:11 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 17 Apr 2014 22:03:11 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20140417215343.GA18833@phdru.name> References: <22741-1397762051-908018@sneakemail.com> <1F992A6C-6C28-4447-A637-487838D7547E@yahoo.com> <20140417215343.GA18833@phdru.name> Message-ID: On Apr 17, 2014, at 14:53, Oleg Broytman wrote: > While I don't think it's a good addition for Python... > > On Thu, Apr 17, 2014 at 02:41:56PM -0700, Andrew Barnert wrote: >> On Apr 17, 2014, at 12:14, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: >> >>> switch_stmt ::= "switch" switch_expr "case" case_expr ":" suite >>> ("case" | "elcase" case_expr ":" suite)* >>> ["else" ":" suite] >>> switch_expr ::= expression >>> case_expr ::= expression_list >> >>> - if case_expr is a tuple, the case suite will be executed if switch_expr is a member of the tuple >> >> So there's no way to switch on a tuple? > > A tuple can be a member of a bigger tuple: > >>>> (1, 'b') in ((1, 'a'), (1, 'b'), (1, 'a', 'b')) > True Well, yeah, but if you want to match (1, 'b') you'd have to write case ((1, 'b'),). Which is bad for static cases, and even worse for dynamic ones. From greg.ewing at canterbury.ac.nz Fri Apr 18 02:44:05 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 Apr 2014 12:44:05 +1200 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: References: Message-ID: <53507555.7020908@canterbury.ac.nz> Andrew Barnert wrote: > There are a few cases where Python deals with this problem by treating tuples > specially (e.g., % formatting), but I don't think anyone wants to extend that > solution. The startswith() and endswith() methods already accept a tuple in place of a string, and require it to be a tuple. So I think it would be entirely reasonable to do the same for replace(). -- Greg From g.rodola at gmail.com Fri Apr 18 13:34:45 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 18 Apr 2014 13:34:45 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter Message-ID: With current socket.sendall() implementation if an error occurs it's impossible to tell how much data was sent. As such I'm wondering whether it would make sense to add a "counter" parameter which gets incremented internally: sent = 0 try: sock.sendall(data, counter=sent) except socket.error as err: priint("only %s bytes were sent" % sent) This would both allow to not lose information on error and avoid keeping track of the total data being sent, which usually requires and extra len() call. E.g. when sending a file: file = open('somefile', 'rb') total = 0 while True: chunk = file.read(8192) if not chunk: break sock.sendall(chunk, counter=total) Thoughts? -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Apr 18 13:52:30 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 18 Apr 2014 21:52:30 +1000 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: References: Message-ID: On Fri, Apr 18, 2014 at 9:34 PM, Giampaolo Rodola' wrote: > With current socket.sendall() implementation if an error occurs it's > impossible to tell how much data was sent. As such I'm wondering whether it > would make sense to add a "counter" parameter which gets incremented > internally: > > sent = 0 > try: > sock.sendall(data, counter=sent) > except socket.error as err: > priint("only %s bytes were sent" % sent) That doesn't work with a straight integer; there's no way for sendall() to modify your integer. It'd need to be some mutable type, or else the new value would have to be returned somewhere. ChrisA From solipsis at pitrou.net Fri Apr 18 14:04:46 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 18 Apr 2014 14:04:46 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter References: Message-ID: <20140418140446.6f8c76ad@fsol> On Fri, 18 Apr 2014 13:34:45 +0200 "Giampaolo Rodola'" wrote: > With current socket.sendall() implementation if an error occurs it's > impossible to tell how much data was sent. As such I'm wondering whether it > would make sense to add a "counter" parameter which gets incremented > internally: > > sent = 0 > try: > sock.sendall(data, counter=sent) > except socket.error as err: > priint("only %s bytes were sent" % sent) > > This would both allow to not lose information on error and avoid keeping > track of the total data being sent, which usually requires and extra len() > call. E.g. when sending a file: > > file = open('somefile', 'rb') > total = 0 > while True: > chunk = file.read(8192) > if not chunk: > break > sock.sendall(chunk, counter=total) Why not simply use send() in such cases? (or, if you want something file-like, call makefile() and then write()) Regards Antoine. From g.rodola at gmail.com Fri Apr 18 14:06:05 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 18 Apr 2014 14:06:05 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: References: Message-ID: On Fri, Apr 18, 2014 at 1:52 PM, Chris Angelico wrote: > On Fri, Apr 18, 2014 at 9:34 PM, Giampaolo Rodola' > wrote: > > With current socket.sendall() implementation if an error occurs it's > > impossible to tell how much data was sent. As such I'm wondering whether > it > > would make sense to add a "counter" parameter which gets incremented > > internally: > > > > sent = 0 > > try: > > sock.sendall(data, counter=sent) > > except socket.error as err: > > priint("only %s bytes were sent" % sent) > > That doesn't work with a straight integer; there's no way for > sendall() to modify your integer. It'd need to be some mutable type, > or else the new value would have to be returned somewhere. > Ouch! It seems you're right, I didn't take that into account. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn at masklinn.net Fri Apr 18 14:12:01 2014 From: masklinn at masklinn.net (Masklinn) Date: Fri, 18 Apr 2014 14:12:01 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: References: Message-ID: <701B4DB6-FF4B-4572-AA1A-91CDF7473469@masklinn.net> On 2014-04-18, at 13:52 , Chris Angelico wrote: > On Fri, Apr 18, 2014 at 9:34 PM, Giampaolo Rodola' wrote: >> With current socket.sendall() implementation if an error occurs it's >> impossible to tell how much data was sent. As such I'm wondering whether it >> would make sense to add a "counter" parameter which gets incremented >> internally: >> >> sent = 0 >> try: >> sock.sendall(data, counter=sent) >> except socket.error as err: >> priint("only %s bytes were sent" % sent) > > That doesn't work with a straight integer; there's no way for > sendall() to modify your integer. It'd need to be some mutable type, > or else the new value would have to be returned somewhere. sendall could just raise a subtype of socket.error with the actually-sent-bytes count attached couldn't it? From g.rodola at gmail.com Fri Apr 18 14:28:22 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 18 Apr 2014 14:28:22 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: <20140418140446.6f8c76ad@fsol> References: <20140418140446.6f8c76ad@fsol> Message-ID: On Fri, Apr 18, 2014 at 2:04 PM, Antoine Pitrou wrote: > On Fri, 18 Apr 2014 13:34:45 +0200 > "Giampaolo Rodola'" > wrote: > > With current socket.sendall() implementation if an error occurs it's > > impossible to tell how much data was sent. As such I'm wondering whether > it > > would make sense to add a "counter" parameter which gets incremented > > internally: > > > > sent = 0 > > try: > > sock.sendall(data, counter=sent) > > except socket.error as err: > > priint("only %s bytes were sent" % sent) > > > > This would both allow to not lose information on error and avoid keeping > > track of the total data being sent, which usually requires and extra > len() > > call. E.g. when sending a file: > > > > file = open('somefile', 'rb') > > total = 0 > > while True: > > chunk = file.read(8192) > > if not chunk: > > break > > sock.sendall(chunk, counter=total) > > Why not simply use send() in such cases? > (or, if you want something file-like, call makefile() and then write()) > > Regards > > Antoine. send() requires to keep track of how much data has actually been sent, hence you need to add some additional logic to resend the missing (tail) part (e.g.: http://hg.python.org/cpython/file/7433f7bce880/Lib/asyncio/unix_events.py#l389 ). That is is fine, but also a bit annoying when you're lazy and grumpy like me. ;-) Actually what would really be useful for sendall() in order to be as "nicer" as possible in terms of usability would be to: 1 - return the number of bytes sent instead of None; that would spare you from using "total += len(data)" on every iteration. 2 - in case of error set a "sent" attribute to the returned (socket.error) exception. I'm not sure how #2 is feasible in practice though nor if it's acceptable to have only certain socket.error exceptions providing extra attributes. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Apr 18 14:35:35 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 18 Apr 2014 14:35:35 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter References: <20140418140446.6f8c76ad@fsol> Message-ID: <20140418143535.7e2c5da1@fsol> On Fri, 18 Apr 2014 14:28:22 +0200 "Giampaolo Rodola'" wrote: > > 2 - in case of error set a "sent" attribute to the returned (socket.error) > exception. If you want to write the logic to re-send the remaining data on error, why not also write the logic to use send() and re-send the remaining data on success? AFAICT, it's the same logic. Regards Antoine. From masklinn at masklinn.net Fri Apr 18 14:50:44 2014 From: masklinn at masklinn.net (Masklinn) Date: Fri, 18 Apr 2014 14:50:44 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: <20140418143535.7e2c5da1@fsol> References: <20140418140446.6f8c76ad@fsol> <20140418143535.7e2c5da1@fsol> Message-ID: <249FF762-2E1D-4C73-84B5-250475A23D64@masklinn.net> On 2014-04-18, at 14:35 , Antoine Pitrou wrote: > On Fri, 18 Apr 2014 14:28:22 +0200 > "Giampaolo Rodola'" > wrote: >> >> 2 - in case of error set a "sent" attribute to the returned (socket.error) >> exception. > > If you want to write the logic to re-send the remaining data on error, > why not also write the logic to use send() and re-send the remaining > data on success? AFAICT, it's the same logic. sendall keeps trying to send stuff until finished or there's an error, so surely there's no point in wrapping *that* in yet an other send loop? But the amount of data which was successfully sent can be useful for reporting, logging, or to check whether complete segments were sent (if the protocol provides such a thing). From g.rodola at gmail.com Fri Apr 18 14:45:28 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 18 Apr 2014 14:45:28 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: <20140418143535.7e2c5da1@fsol> References: <20140418140446.6f8c76ad@fsol> <20140418143535.7e2c5da1@fsol> Message-ID: On Fri, Apr 18, 2014 at 2:35 PM, Antoine Pitrou wrote: > On Fri, 18 Apr 2014 14:28:22 +0200 > "Giampaolo Rodola'" > wrote: > > > > 2 - in case of error set a "sent" attribute to the returned > (socket.error) > > exception. > > If you want to write the logic to re-send the remaining data on error, > why not also write the logic to use send() and re-send the remaining > data on success? AFAICT, it's the same logic. > > Regards > > Antoine. > Tipically in case of error on sendall() you don't want to resend anything because most of the times it's a disconnection error (and never, say, a "retry" error such as EAGAIN). The use case I'm thinking about, and which would probably be the most common one, is to figure out how much data was sent (via exc.sent or something), add it to "total", reconnect and resume the transfer from where you left. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Apr 18 16:40:25 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Apr 2014 07:40:25 -0700 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: References: Message-ID: Never gonna happen, for all the reasons brought up. Give it up. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From 7vsfeu4pxg at snkmail.com Fri Apr 18 16:42:52 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Fri, 18 Apr 2014 16:42:52 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> Message-ID: <4588-1397832212-797003@sneakemail.com> On 17 April 2014 21:31, Guido van Rossum guido-at-python.org | python-ideas-at-python.org| wrote: > I don't want to discourage you too much, but I think that adding a switch > statement comes *very* low on the list of improvements we would like to > make in Python 3.5. > > We should probably focus on speed (or aspects of it, like startup), > language features that help porting Python 2 code to it (e.g. bytes > formatting), and things that improve the user experience of getting started > with Python on a new machine (e.g. pip/venv).. Or perhaps stdlib issues > like an asyncio-infused variation of WSGI. > I completely agree. > > I've probably missed a few focus areas, but I still very much doubt we'll > be adding a switch statement -- it's a "sexy" language design issue (like > anonymous functions) but that's not what will help Python compete. > Well, no, it will not be a "killer feature". But if you intend it's "sexy" because it's a syntax sugar, even if I'm not much experienced, I respectfully disagree. A switch-case statement is more DRY than an if-elif chain, in its use environment. Deltas are only quicker to code and potentially less DRY. On 17 April 2014 23:41, Andrew Barnert abarnert-at-yahoo.com | > python-ideas-at-python.org| <3en9kh2cbt at sneakemail.com> wrote: > Any reason to use the keywords that only exist in C-family languages when > the functionality isn't actually like C? > Well, it's the same for the "for" statement. > - if case_expr is not a tuple, the case suite will be executed if > switch_expr == case_expr > > - if a case_expr is checked, any subsequent elcase statements are > skipped, and the next case statement is performed, of there's one. This is > completely identical to if - elif. > > While you can figure out what this means with a bit of work, it seems to > add a lot of conceptual complexity, and I can't think of any other language > that does this. > Bash case statement is similar. IMHO this syntax is very simple, simpler than C-like switch statements. Why do you think it is complex? why not consider features that other languages do have, like pattern > matching cases? > I do not know much about pattern matching. Using my little knowledge, Python have recursive functions and regular expressions. You could also use generator expressions with my switch statement proposal. Symbolic pattern is powerful, but it's a bit too much for this little proposal. if you want to match (1, 'b') you'd have to write case ((1, 'b'),). Which > is bad for static cases, and even worse for dynamic ones. > Yes, it's not so elegant, but I do not think that it will really matters in real world usage. What do you mean exactly with "dynamic cases"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Fri Apr 18 17:03:07 2014 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 18 Apr 2014 12:03:07 -0300 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <22741-1397762051-908018@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> Message-ID: On 17 April 2014 16:14, Lucas Malor <7vsfeu4pxg at snkmail.com> wrote: > A simpler example: > > switch tarot case 0: > card = "Fool" > elcase 1: > card = "Alan Moore" > elcase 2: > card = "High Priestess" > It may be just me, but I fail - in a complete manner - to see how this syntax can offer any improvement on current if/elif chains. It seems to differ from that only by reducing the explicitness, and the flexibility of the test condition that can be used in any of the subconditions, No, sorry, unlike others have mentioned, I don't find this particularly "sexy". "Clumky" is more like it: if tarot ==0: card = "Fool" elif tarot == 1: card = "Alan Moore" elif 2 < tarot < 22: # expressiveness card = "Surprise" else: raise ValueError The implicit "in" operation case is even more weird. js -><- From skip at pobox.com Fri Apr 18 17:22:14 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 18 Apr 2014 10:22:14 -0500 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> Message-ID: On Fri, Apr 18, 2014 at 10:03 AM, Joao S. O. Bueno wrote: > It may be just me, but I fail - in a complete manner - to see how this > syntax can offer any > improvement on current if/elif chains. I haven't been following this thread closely, so I don't know if Lucas has mentioned more than syntax, however... In other languages, the semantics of the switch statement allow the compiler to generate more efficient code. Instead of testing each branch of the if statement in succession looking for a match, you evaluate the switch expression once, then use it as an index into a jump table of some sort which points directly to the matching case label. If nothing matches, you jump to the default branch (if one was defined) or just jump over the entire switch statement (if not). Skip From skip at pobox.com Fri Apr 18 17:26:57 2014 From: skip at pobox.com (Skip Montanaro) Date: Fri, 18 Apr 2014 10:26:57 -0500 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> Message-ID: I should also mention that the use of a jump table depends on the use of case labels which can be evaluated at compile time. This might make it challenging (or impossible) for this scheme to be implemented in Python. Skip From g.rodola at gmail.com Fri Apr 18 18:21:03 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 18 Apr 2014 18:21:03 +0200 Subject: [Python-ideas] socket.sendall() "counter" parameter In-Reply-To: References: Message-ID: On Fri, Apr 18, 2014 at 4:40 PM, Guido van Rossum wrote: > Never gonna happen, for all the reasons brought up. Give it up. :-) > > -- > --Guido van Rossum (python.org/~guido) > Fair enough (I was not convinced either). =) -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Apr 18 18:51:26 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 18 Apr 2014 09:51:26 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <4588-1397832212-797003@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> Message-ID: <5E5E6F9B-DEE8-4D4C-869F-0E848CC88EBF@yahoo.com> On Apr 18, 2014, at 7:42, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > >> On 17 April 2014 23:41, Andrew Barnert abarnert-at-yahoo.com |python-ideas-at-python.org| <3en9kh2cbt at sneakemail.com> wrote: >> Any reason to use the keywords that only exist in C-family languages when the functionality isn't actually like C? > > Well, it's the same for the "for" statement. No it isn't. First, the "for ... in" keywords are not the same as just "for" and a bunch of parens and semicolons. If you just mean the word "for" itself, the exact same word is used in a wide variety of languages that have nothing to do with C, and some of them use it with the same meaning as Python. By contrast, most languages that are not related to C don't spell this statement "switch ... case ..."; the most common is probably "case ... of ..." or "case ... in ...". Also, notice that if you try to read the switch statement, or your Python version, as English, it's nonsense. Compare that to for, while, if, try, etc. While there are some oddities (like condensing "else" and "if" into "elif"), as a general rule they make sense. That's not even remotely true with a C for statement. >> > - if case_expr is not a tuple, the case suite will be executed if switch_expr == case_expr >> > - if a case_expr is checked, any subsequent elcase statements are skipped, and the next case statement is performed, of there's one. This is completely identical to if - elif. >> >> While you can figure out what this means with a bit of work, it seems to add a lot of conceptual complexity, and I can't think of any other language that does this. > > > Bash case statement is similar. Bash doesn't have separate "case" and "elcase" cases. After one case is done, the rest are skipped, just as in most other languages. > IMHO this syntax is very simple, simpler than C-like switch statements. Why do you think it is complex? C cases always fall through to the next case. Almost every other language only executes a single case. I don't see how skipping over any elcase but falling through to the next case is in any way simpler than C. >> why not consider features that other languages do have, like pattern matching cases? > > I do not know much about pattern matching. Well, then at least look at the limited form of pattern matching Python has in the for and assignment statements and parameter matching, and maybe look at how pattern matching is used with case statements in other languages; don't try to suggest language designs based on guesses. > Using my little knowledge, Python have recursive functions and regular expressions. ... and? Are you suggesting that if the switch expression is a string and the case expression a compiled regex you could automatically call match instead of testing for equality? If not, how is having regexp even relevant here? And how are recursive functions relevant? > You could also use generator expressions with my switch statement proposal. How? Where? A generator expression is equal to anything except itself, and doesn't contain anything. > Symbolic pattern is powerful, but it's a bit too much for this little proposal. I don't know what you mean by "symbolic pattern" here. >> if you want to match (1, 'b') you'd have to write case ((1, 'b'),). Which is bad for static cases, and even worse for dynamic ones. > > Yes, it's not so elegant, but I do not think that it will really matters in real world usage. Really? Have you never wanted to switch on the results of a function that returns two values? > What do you mean exactly with "dynamic cases"? Exactly what it sounds like: when the case value is a variable set somewhere else in the program, possibly by some program calling my library or even selected by user input or config, so at the point of the switch statement you (meaning a reader of the code) have no idea whether the value is a tuple; it's just some variable or lookup expression or the like. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-ideas at hmmz.org Fri Apr 18 20:04:21 2014 From: dw+python-ideas at hmmz.org (David Wilson) Date: Fri, 18 Apr 2014 19:04:21 +0100 Subject: [Python-ideas] Python-specific Travis CI Message-ID: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> Hi there, In my travails to get py-lmdb (github.com/dw/py-lmdb) into a state I can depend on, I've been exploring continuous integration options since sometime mid last year, in the hopes of finding the simplest option that allows the package to be tested on all sensible versions of Python that I might bump into in a commercial environment. It seems this simple task currently yields one solution - Travis CI, except that by default Travis supports neither Windows, nor any version of Python older than about 3 months, without requiring serious manual configuration and maintenance. The library just about has a working Travis CI at this point ??although it is incredibly ugly. Testing against all recent versions of Python (by "recent" I mean "anything I've seen and might reasonably expect to bump into while consulting, i.e. since 2.5") requires around 14 Debian packages to be installed, followed by installation of compatible versions of easy_install and Pip for each interesting Python version. The result just about works, but this week I'm again finding that it doesn't quite go far enough, as I begin working on a Windows binary distribution for my package. Windows is almost the same problem all over again - testing supported/sensible extension module builds requires 14 binary installations of Python (one for each of 32 bit and 64 bit), 3 distinct versions of Microsoft Visual Studio (that I'm aware of so far), and a compatible version of Windows to run it all on. So the work done to support CI on Travis more or less isn't enough. It seems the best option now is a custom Jenkins setup, and acquiring licenses for all the Microsoft software necessary to build the package. You may wonder why one might care about so many old Python versions in weird and exotic environments - to which I can't really give a better answer than that I've lost so much time working with packages that suffer from platform/version compatibility issues, that I'd like to avoid it in my own code. While it isn't so true in web development, there are pretty huge installations of Python around the world, many of which aren't quite near migrating to 2.6, never mind 3.4. This can happen for different reasons, from maintaining private forks of Python (perhaps in a post-audited state, where another audit simply isn't in the budget), through to having thousands of customers who have written automation scripts for their applications using an older dialect. In any case it's enough to note these installations do and will continue to exist, than to question why they exist.. So in the coming months I'll probably end up configuring a Jenkins, porting the Travis CI scripts over to it to a private Ubuntu VM configured as a Jenkins runner, and setting up a new Windows VM as a second runner, but it seems entirely wasteful that so much effort should be made for just one project. And that leads to thinking about a Python-specific CI service, that could support builds similar to Travis CI, except for all configurations of Python still in active use. One aspect of such a service that is particularly interesting, is the ability to centralize management of the required licenses ? the PSF has a much greater reputation, and is much more likely to be granted a blanket license covering a CI service for all open source Python code, than each individual project is by applying individually. Licensing aside, having a 0-cost option for any Python project to be tested would improve the quality of our ecosystem overall, especially in the middle of the largest transition to date - the 3.x series. Following from the ecosystem improvement angle, there might be no reason a Python-specific CI service could not automatically download, build and py.test every new package entering PyPI, producing as side effect scoreboards similar to the Python 3 Wall of Superpowers (http://python3wos.appspot.com/). This would make a sound base for building future improvements for PyPI - for example a lint service, verifying "setup.py" is not writing to /etc/passwd, or uploaded eggs aren't installing a new SVCHOST.EXE to %SystemDir%. It could even make sense to have the hypothetical service produce signed, "golden" artefacts for upload to PyPI - binaries built from source in a trusted environment, free of any Bitcoin ransom-ware surreptitiously inserting itself into the bdist_egg ZIP file. There are many problems with providing a CI service, for example, the handling of third party commercial libraries required during the build process. An example of this might be producing a build of cx_Oracle, where Oracle's proprietary SDK is a prerequisite. Still, it would not require many resources, or even very much code, to provide a service that could minimally cover at least basic needs for purely open source software, and so the idea appeals to me. Any thoughts? David From donald at stufft.io Fri Apr 18 21:07:18 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 18 Apr 2014 15:07:18 -0400 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> Message-ID: <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> On Apr 18, 2014, at 2:04 PM, David Wilson wrote: > Hi there, > > In my travails to get py-lmdb (github.com/dw/py-lmdb) into a state I can depend on, I've been exploring continuous integration options since sometime mid last year, in the hopes of finding the simplest option that allows the package to be tested on all sensible versions of Python that I might bump into in a commercial environment. > > It seems this simple task currently yields one solution - Travis CI, except that by default Travis supports neither Windows, nor any version of Python older than about 3 months, without requiring serious manual configuration and maintenance. > > The library just about has a working Travis CI at this point ? although it is incredibly ugly. Testing against all recent versions of Python (by "recent" I mean "anything I've seen and might reasonably expect to bump into while consulting, i.e. since 2.5") requires around 14 Debian packages to be installed, followed by installation of compatible versions of easy_install and Pip for each interesting Python version. Travis comes with 2.6, 2.7, 3.2, and 3.3 as well as PyPy 2.2.1 out of the box and I?ve got a PR merged there which is waiting on a deploy which will expand that out to also include 3.4 (as well as making a command line available that will build 2.4, 2.5, Jython, and others (Basically the latest versions of the above are installed, and you can use ``python-build`` to build anything located here https://github.com/yyuu/pyenv/tree/master/plugins/python-build/share/python-build). And honestly with the new PR I?ve laid out in Travis adding any older versions to be supported out of the box is simple and the biggest barrier is likely convincing the Travis folks they are needed (For example, if you?re distributing on PyPI the number of 2.5 or 3.1 users are *tiny* https://caremad.io/blog/a-look-at-pypi-downloads/). > > The result just about works, but this week I'm again finding that it doesn't quite go far enough, as I begin working on a Windows binary distribution for my package. > > Windows is almost the same problem all over again - testing supported/sensible extension module builds requires 14 binary installations of Python (one for each of 32 bit and 64 bit), 3 distinct versions of Microsoft Visual Studio (that I'm aware of so far), and a compatible version of Windows to run it all on. As far as I?m aware the Travis-CI folks are working on Windows support but there is currently no ETA on it as they have to figure out licensing concerns and the like. > > So the work done to support CI on Travis more or less isn't enough. It seems the best option now is a custom Jenkins setup, and acquiring licenses for all the Microsoft software necessary to build the package. One thing to look out for here, is that you can?t test arbitrary PRs easily with this solution without whitelisting contributors because Travis runs a PR test in an isolated VM instance, whereas Jenkins runs it on a shared machine, and this shared machine through the jenkins API has more or less full control of the entire jenkins cluster. > > > You may wonder why one might care about so many old Python versions in weird and exotic environments - to which I can't really give a better answer than that I've lost so much time working with packages that suffer from platform/version compatibility issues, that I'd like to avoid it in my own code. > > While it isn't so true in web development, there are pretty huge installations of Python around the world, many of which aren't quite near migrating to 2.6, never mind 3.4. This can happen for different reasons, from maintaining private forks of Python (perhaps in a post-audited state, where another audit simply isn't in the budget), through to having thousands of customers who have written automation scripts for their applications using an older dialect. > > In any case it's enough to note these installations do and will continue to exist, than to question why they exist.. > > > So in the coming months I'll probably end up configuring a Jenkins, porting the Travis CI scripts over to it to a private Ubuntu VM configured as a Jenkins runner, and setting up a new Windows VM as a second runner, but it seems entirely wasteful that so much effort should be made for just one project. > > And that leads to thinking about a Python-specific CI service, that could support builds similar to Travis CI, except for all configurations of Python still in active use. One aspect of such a service that is particularly interesting, is the ability to centralize management of the required licenses ? the PSF has a much greater reputation, and is much more likely to be granted a blanket license covering a CI service for all open source Python code, than each individual project is by applying individually. > > Licensing aside, having a 0-cost option for any Python project to be tested would improve the quality of our ecosystem overall, especially in the middle of the largest transition to date - the 3.x series. > > Following from the ecosystem improvement angle, there might be no reason a Python-specific CI service could not automatically download, build and py.test every new package entering PyPI, producing as side effect scoreboards similar to the Python 3 Wall of Superpowers (http://python3wos.appspot.com/). This would make a sound base for building future improvements for PyPI - for example a lint service, verifying "setup.py" is not writing to /etc/passwd, or uploaded eggs aren't installing a new SVCHOST.EXE to %SystemDir%. I?ve thought about working on something like this for PyPI, one of the big barriers there being that there is no central way to define what versions you support, what OSs, what required _external_ dependencies you have, or even what your test runner is (besides setup.py test which is far from standard or ubiquitous). > > It could even make sense to have the hypothetical service produce signed, "golden" artefacts for upload to PyPI - binaries built from source in a trusted environment, free of any Bitcoin ransom-ware surreptitiously inserting itself into the bdist_egg ZIP file. A build farm for Wheels is something else that I?d love to do. > > There are many problems with providing a CI service, for example, the handling of third party commercial libraries required during the build process. An example of this might be producing a build of cx_Oracle, where Oracle's proprietary SDK is a prerequisite. > > Still, it would not require many resources, or even very much code, to provide a service that could minimally cover at least basic needs for purely open source software, and so the idea appeals to me. I think you?re vastly underestimating the complexity in machine resources, code, and time. For security purposes you are more or less required to use throw away VMs which get destroyed after a single use. The sanest way of doing this is to manage a pool of VM workers, pull one off the pool, run tasks on it, and then destroy it and let the pool spawn a new one. This puts a bottleneck on how many tests you can run both in how large your pool is, and in how fast your VMs boot. Look at a project like https://travis-ci.org/pyca/cryptography which builds 40 different configurations for each ?build? each taking about 8 minutes each. So if you have a pool of 5 VMs, you can run 5 tests at a time, so that splits that 40 into 5 chunks, and you get roughly 64 minutes to process 40 builds, plus we?ll say our machines boot in roughly 60s (a little fast for cloud servers, but not unreasonable), so there?s an additional 4-5 minutes just in booting. So roughly an hour and 10 minutes for a single test run if with 5 VMs for just a single project. (In reality Cryptography has more than 40 builds because it uses Travis and jenkins together to handle things travis doesn?t). So the machine cost is high, you?re probably looking at let?s just say a worker pool of 20 (though I think to actually replace Travis CI it?d need to be much higher) of roughly 4GB machines (last I recall travis machines were 3gig and change) which comes out to roughly 1k to 2k in server costs just for that pool. Add into that whatever support machines you would need (queues, web nodes, database servers, etc) you?re probably looking in the 2-4k range just for the servers once all was said and done. I believe the code cost would also be fairly high. There isn?t really an off the shelf solution that is going to work for this. You?ve mentioned Jenkins, however Jenkins does not scale well at all (look at the Openstack infra for all the things they?ve invented around Jenkins to try and scale it). There is a good chance some things could be reused from Openstack but I?m fairly sure some of it is specific enough to Openstack that it?d still require a decent amount of coding to make it generic enough to work. I also think the time investment would be high outside of the effort to build it. I hang out in the #openstack-infra channel and I can see how much effort goes into maintaing that infra (and i?m sure it?s similar to travis-ci). This means that it?s likely not going to be something we can just set it up and forget about it and will require an active infrastructure team to handle it. Now Python happens to have one of those, but we?re mostly volunteers with part times for on call stuff (who also volunteer for other stuff too!) and this would be a significant increase I believe in that work load. > > Any thoughts? > > > David > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ It?s my personal opinion that a sort of ?test during development? CI system like Travis is not something that making Python specific is very useful in the long run. Travis is open source and they are incredibly open to things that will make Python a more first class citizen there (disclaimer: I?m friends with them and I?ve helped add the Python support to start with, and then improve it over time). One obvious flaw in this is that Travis only supports github while there are others on Bitbucket or even their own hosting, and for those people this idea might be much more attractive. I do think there is maybe room for a ?release testing? system and I definitely think there is room for a build farm (Which is a much smaller scale since the whole of PyPI pushes releases far less often than push commits to branches or make PRs or what have you). Just my 2 cents! ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ncoghlan at gmail.com Fri Apr 18 21:43:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Apr 2014 15:43:56 -0400 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> Message-ID: On 18 April 2014 15:07, Donald Stufft wrote: > On Apr 18, 2014, at 2:04 PM, David Wilson wrote: >> In my travails to get py-lmdb (github.com/dw/py-lmdb) into a state I can depend on, I've been exploring continuous integration options since sometime mid last year, in the hopes of finding the simplest option that allows the package to be tested on all sensible versions of Python that I might bump into in a commercial environment. >> >> It seems this simple task currently yields one solution - Travis CI, except that by default Travis supports neither Windows, nor any version of Python older than about 3 months, without requiring serious manual configuration and maintenance. And has the added downside of only working with GitHub (alas, the demise of Shining Panda CI rather narrowed the field of non-GitHub reliant CI options). Anyway, building out CPython's continuous integration infrastructure is indeed on my todo list, but that will be a long term project, and focus on CPython first (the new core-workflow list will be the home of any discussions about that, although it ay bleed over into the infrastructure SIG at times). Doing CI well is a genuinely hard problem, and that applies even when you trust the software being tested. Testing untrusted software ups the difficulty considerably. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From merwok at netwok.org Fri Apr 18 21:49:43 2014 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 18 Apr 2014 15:49:43 -0400 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: Message-ID: <535181D7.1020505@netwok.org> Hi, Le 15/04/2014 22:12, Nick Coghlan a ?crit : > Something that came up during PyCon was the idea of a "py" script that > brought "Python Launcher for Windows" explicit version dispatch to Linux. As a Debian end user and Python developer, I don?t have a need for this; Unixen have shebangs and naming conventions (more or less similar across OSes) for co-installed versions of Python. (?Explicit dispatch? and ?cross-platform invocation? don?t tell me much.) I went back and read the beginning of PEP 397 again to find use cases: Abstract This PEP describes a Python launcher for the Windows platform. A Python launcher is a single executable which uses a number of heuristics to locate a Python executable and launch it with a specified command line. Rationale Windows provides "file associations" so an executable can be associated with an extension, allowing for scripts to be executed directly in some contexts (eg., double-clicking the file in Windows Explorer.) Until now, a strategy of "last installed Python wins" has been used and while not ideal, has generally been workable due to the conservative changes in Python 2.x releases. As Python 3.x scripts are often syntactically incompatible with Python 2.x scripts, a different strategy must be used to allow files with a '.py' extension to use a different executable based on the Python version the script targets. This will be done by borrowing the existing practices of another operating system - scripts will be able to nominate the version of Python they need by way of a "shebang" line, as described below. It seems to me the problem is defined as specific to Windows, and the solution takes inspiration from other operating systems. I think a new rationale explaining why bring back that solution to these other OSes is needed. Regards From ncoghlan at gmail.com Sat Apr 19 00:27:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 Apr 2014 18:27:04 -0400 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: <535181D7.1020505@netwok.org> References: <535181D7.1020505@netwok.org> Message-ID: On 18 April 2014 15:49, ?ric Araujo wrote: > > It seems to me the problem is defined as specific to Windows, and the > solution takes inspiration from other operating systems. I think a new > rationale explaining why bring back that solution to these other OSes is > needed. It would be about removing the current cross-platform discrepancy in the instructions at https://docs.python.org/3/installing/#work-with-multiple-versions-of-python-installed-in-parallel Not a high priority for me personally, but I figured it was worth mentioning in case it captured someone's interest. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From 7vsfeu4pxg at snkmail.com Sat Apr 19 01:08:58 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Sat, 19 Apr 2014 01:08:58 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> Message-ID: <5941-1397862578-751436@sneakemail.com> On 18 April 2014 16:58, Ed Kellett edk141-at-gmail.com | python-ideas-at-python.org| wrote: > case foo(): > > would have to become > > case (foo(),): > > to work as expected when foo() returned a tuple, which would mean > wrapping things in 1-tuples whenever you wanted to reliably match a > case that is determined dynamically. > To obviate this, instead of tuples case could check membership of CaseTuples. CaseTuple will be a class that extends tuple, identical to it, and a case expression list with commas will yield a CaseTuple. To use other iterables as case expression list you'll have to unpack them, or they will be matched for equality. On 18 April 2014 17:03, Joao S. O. Bueno jsbueno-at-python.org.br | python-ideas-at-python.org| <0ucjz7dbjt at sneakemail.com> wrote: > It may be just me, but I fail - in a complete manner - to see how this > syntax can offer any > improvement on current if/elif chains. It seems to differ from that > only by reducing > the explicitness, and the flexibility of the test condition that can > be used in any > of the subconditions > It's more simple to use when you can use it, as switch statement in the other languages. And it somewhat adheres to the DRY principle: why repeat the subject? If I'm checking what type of tarot card I have, why should I repeat every time I'm trying to identify a tarot card? I also thought about a syntax like this: "case" comparator case_expr ":" but IMHO it's too verbose for the typical uses of switch. If you want flexibility, you can always use if-elif. @Skip Montanaro: yes, switch statement is used in C also for code optimization. Frankly, I think this aspect is unimportant for CPython in the present time. On 18 April 2014 18:51, Andrew Barnert abarnert-at-yahoo.com | python-ideas-at-python.org| <3en9kh2cbt at sneakemail.com> wrote: > No it isn't. First, the "for ... in" keywords are not the same as just > "for" and a bunch of parens and semicolons. > My propose has elcase, that is not present in other languages. Also, notice that if you try to read the switch statement, or your Python > version, as English, it's nonsense. > Yes, the fact a case will behave differently for tuples and non-tuples will be difficult to translate in English. I think that with CaseTuple proposal it will be more understandable, since you have to explicitly unpack an iterable, or use elements separated by commas. Bash doesn't have separate "case" and "elcase" cases. After one case is > done, the rest are skipped, just as in most other languages. > But it has ;& and ;;& similarly to break and continue of C, that it's equivalent to case and elcase of my proposal. I don't see how skipping over any elcase but falling through to the next > case is in any way simpler than C. Well, because it's coherent with if-elif. See my last example in my first message. Well, then at least look at the limited form of pattern matching Python has > in the for and assignment statements and parameter matching, and maybe look > at how pattern matching is used with case statements in other languages; > don't try to suggest language designs based on guesses. > Excuse me? I know list comprehensions, lambdas and argument unpacking. And I do not think you can see what I do before I post a message. If so, you could see me googling before writing about something that I don't know very well or I don't remember very well. So don't guess about what I do or not do or know and not know, thank you. About pattern matching in the for statement, I really don't know what they are. .... and? Are you suggesting that if the switch expression is a string and > the case expression a compiled regex you could automatically call match > instead of testing for equality? If not, how is having regexp even relevant > here? And how are recursive functions relevant? > I'm suggesting to use if-elif with re module, if you want to use regular expression, and to use recursive functions if you want... recursive functions. To be more clear, IMHO switch-case is useful if it's simple. A generator expression is equal to anything except itself, and doesn't > contain anything. > You can convert it to an iterable. Probably an overkill, but you can do it. I don't know what you mean by "symbolic pattern" here. > For what I know (not too much), in Mathematica pattern matching can be used for symbols, and symbols can be used as identifiers: https://reference.wolfram.com/mathematica/guide/Patterns.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Sat Apr 19 02:14:35 2014 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 18 Apr 2014 20:14:35 -0400 Subject: [Python-ideas] py launcher for Linux? In-Reply-To: References: <535181D7.1020505@netwok.org> Message-ID: <5351BFEB.8090509@trueblade.com> On 4/18/2014 6:27 PM, Nick Coghlan wrote: > On 18 April 2014 15:49, ?ric Araujo wrote: >> >> It seems to me the problem is defined as specific to Windows, and the >> solution takes inspiration from other operating systems. I think a new >> rationale explaining why bring back that solution to these other OSes is >> needed. > > It would be about removing the current cross-platform discrepancy in > the instructions at > https://docs.python.org/3/installing/#work-with-multiple-versions-of-python-installed-in-parallel > > Not a high priority for me personally, but I figured it was worth > mentioning in case it captured someone's interest. I'm sort of interested. I'm still working on the semantic differences between the Windows and Unix versions. Parsing the .ini files in C or sh/awk/sed is definitely a hassle. I can't decide if we should assume there's a python executable on the PATH or not. Windows has the benefit of GetPrivateProfileString to do the heavy lifting there. Eric. From ron3200 at gmail.com Sat Apr 19 03:40:35 2014 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 18 Apr 2014 21:40:35 -0400 Subject: [Python-ideas] str.find() and friends support a lists of inputs In-Reply-To: <20140418022222.GF28400@ando> References: <20140418022222.GF28400@ando> Message-ID: On 04/17/2014 10:22 PM, Steven D'Aprano wrote: > On Thu, Apr 17, 2014 at 09:31:24PM -0400, Terry Reedy wrote: >> >On 4/17/2014 2:52 PM, Alex Rodrigues wrote: >>> > >It's a fairly common problem to want to .find() or .replace() or .split() >>> > >any one of multiple characters. >>> > >Currently the go to solution are: >> > >> >For replace, you left out the actual solution. >> > >>>>> > >>>telnum = '(800), 555-1234' >>>>> > >>>telnum.translate(str.maketrans('','','(- ,)')) >> >'8005551234' > That solution only works when you want to replace single characters. It > doesn't help to replace generic substrings: > > "That's one small step for a man, one giant leap for mankind.".replace( > 'man', 'person') > > Naively looping over your input may not work. > > for term in (a, b, c): > s = s.replace(term, x) > > > is not always the same as doing the replacements in a single pass. It > seems like it ought to be the same, until you run into a situation like > this: > > py> recipe = "Add one capsicum to the stew..." > py> for term in ("capsicum", "chilli pepper", "pepper"): > ... recipe = recipe.replace(term, "bell pepper") > ... > py> print(recipe) > Add one bell bell pepper to the stew... > > > Oops! If the search terms are not known until runtime, you may have a > lot of difficulty doing the replacements in an order that doesn't cause > problems like this. There are ways around this problem, but they're > tricky to get right. Possible start by allowing the sep argument for str.partition accept a tuple. recipe, _recipe = "", recipe while _recipe: head, sep, _recipe = _recipe.partition( ("capsium", "Chilli pepper", "pepper")) if sep: recipe = "".join([recipe, head, "bell pepper"]) else: recipe = "".join([recipe, head]) Then maybe the other methods can use str.partition to do the work. Cheers, Ron From vernondcole at gmail.com Sat Apr 19 04:33:32 2014 From: vernondcole at gmail.com (Vernon D. Cole) Date: Fri, 18 Apr 2014 20:33:32 -0600 Subject: [Python-ideas] py launcher for Linux? Message-ID: > Date: Fri, 18 Apr 2014 18:27:04 -0400 > From: Nick Coghlan > > On 18 April 2014 15:49, ?ric Araujo wrote: > > > > It seems to me the problem is defined as specific to Windows, and the > > solution takes inspiration from other operating systems. I think a new > > rationale explaining why bring back that solution to these other OSes is > > needed. > > It would be about removing the current cross-platform discrepancy in > the instructions at > > https://docs.python.org/3/installing/#work-with-multiple-versions-of-python-installed-in-parallel > > Not a high priority for me personally, but I figured it was worth > mentioning in case it captured someone's interest. > > Cheers, > Nick. > +1 I am a switch hitter. I spend almost as much time on Windows as on Linux, and to keep myself from being completely confused, I make the environments as similar as possible. The most frequently-used utility on my Windows box must be "ls.bat" (which runs "dir"). It is true that the "real reason" for py.exe was to enable #! processing, and that works wonderfully -- mine also launches IronPython, Jython, PyPy, and (just to prove a point) Perl. [Trivia: hello_world.pl runs perfectly in Python.] But, the "py" command-line command is really habit forming. For weeks now, "py: command not found" has been haunting me. Somehow the name "Python Launcher for Windows for Linux" sounds wrong, but I want one. It works so well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Apr 19 05:12:24 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 19 Apr 2014 13:12:24 +1000 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <5941-1397862578-751436@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> Message-ID: <20140419031221.GI28400@ando> On Sat, Apr 19, 2014 at 01:08:58AM +0200, Lucas Malor wrote: > On 18 April 2014 17:03, Joao S. O. Bueno jsbueno-at-python.org.br | > python-ideas-at-python.org| <0ucjz7dbjt at sneakemail.com> wrote: > > > It may be just me, but I fail - in a complete manner - to see how this > > syntax can offer any > > improvement on current if/elif chains. It seems to differ from that > > only by reducing > > the explicitness, and the flexibility of the test condition that can > > be used in any > > of the subconditions > > > > It's more simple to use when you can use it, as switch statement in the > other languages. And it somewhat adheres to the DRY principle: why repeat > the subject? If I'm checking what type of tarot card I have, why should I > repeat every time I'm trying to identify a tarot card? I think that is a misunderstanding of the DRY principle. I'll explain further below, but even if the idea is to avoid writing anything twice, the case syntax fails. Consider your example: switch tarot case 0: card = "Fool" elcase 1: card = "Alan Moore" elcase 2: card = "High Priestess" Here, you are repeating "elcase" and "card =" each time, so you are still repeating yourself. We can avoid that by using the lookup table approach: table = {0: "Fool", 1: "Alan Moore", 2: "High Priestess", ...} card = table[tarot] Now that truly is a DRY solution! I think that your interpretation of DRY does not match the intention of the people who invented it. DRY is not (as I understand it) concerned with trivial, mechanical duplication like chained if...elif: if tarot == 0: ... elif tarot == 1: ... elif tarot == 2: ... and objecting to chained if...elif as a DRY-violation is, I believe, a misunderstanding of DRY. I'm going to quote Dave Thomas, co-inventer of the DRY principle: Most people take DRY to mean you shouldn't duplicate code. *That's not its intention.* [emphasis added] The idea behind DRY is far grander than that. DRY says that every piece of system knowledge should have one authoritative, unambiguous representation. Every piece of knowledge in the development of something should have a single representation. A system's knowledge is far broader than just its code. It refers to database schemas, test plans, the build system, even documentation. http://www.artima.com/intv/dry.html DRY is also known as "Single Source Of Truth", which is perhaps a better name, since it emphasises the fact that there is a single canonical source of each piece of knowlege in the system, rather than putting the emphasis on mere mechanical duplication. In the case of your tarot example, it is *not* a violation of DRY, because the various elif lines are not *sources* of knowledge which may contradict each other. The worst that will happen is that if you change the variable name "tarot" to something else, your code will fail with a name error. DRY is not about reducing the amount of mechanical edits you do when you rename a variable. # Single source of truth for the name of the variable is its binding: tarotcard = draw_card() # was "tarot" if tarot == 0: # now fails ... Subsequent lines merely *use* the name, they don't act as potential sources of knowledge which could contract each other. DRY doesn't really have much to say about variable names, except perhaps "don't use the same name (in the same namespace) for different things", so if your motive in introducing a switch/case statement is to DRY, I think you need a better motive. One motive I'd like to see is, could a switch/case statement be used to automate table lookups? Two problems -- albeit mild ones -- with the table lookup idiom is that the table lives longer than needed, and it puts the table in the wrong place. This motivated Nick to suggest a "where" block: card = table[tarot] where: table = {0: "Fool", 1: "Alan Moore", 2: "High Priestess", ...} solving both problems at once: the table is declared only when needed, not before hand, and does not exist outside of the block. A third problem, not solved by Nick's "where", is that the table requires every case's value ahead of time, whether it will be needed or not. When the values are constants, that's not a big deal, but they might be expensive expressions. Perhaps there is some way to optimize a case statement so as to avoid these disadvantages of the table lookup idiom. To me, that is a reasonable motive worth chasing. I wouldn't bother with a case statement unless it was more efficient than a chain of if...elif. > @Skip Montanaro: yes, switch statement is used in C also for code > optimization. Frankly, I think this aspect is unimportant for CPython in > the present time. Then I think we're not going to agree on the use or need for switch. -- Steven From abarnert at yahoo.com Sat Apr 19 06:54:56 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 18 Apr 2014 21:54:56 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <5941-1397862578-751436@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> Message-ID: On Apr 18, 2014, at 16:08, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > > On 18 April 2014 18:51, Andrew Barnert abarnert-at-yahoo.com |python-ideas-at-python.org| <3en9kh2cbt at sneakemail.com> wrote: >> No it isn't. First, the "for ... in" keywords are not the same as just "for" and a bunch of parens and semicolons. > > My propose has elcase, that is not present in other languages. I can't understand why you're still not getting it, but I'll try again. Most languages' case statements are of the form "case ... in ...", sometimes substituting "of" or punctuation in place of "in". C and its derivatives instead use "switch ... case ...". By using the unique C syntax (or a minor variation on it, which won't even be visible in any statement without an elcase) without the C semantics, you're creating a misleading parallel, and for no benefit that I can see. >> Also, notice that if you try to read the switch statement, or your Python version, as English, it's nonsense. > > Yes, the fact a case will behave differently for tuples and non-tuples will be difficult to translate in English. No, that's not at the point. Consider: for ch in mystring: process(ch) You can read the first one aloud and it sounds like an English sentence. Maybe a slightly stilted sentence, the kind of thing you'd hear from a mathematician speaking technically or someone speaking English as a foreign language, but clearly recognizable and understandable to anyone who understands English. And this generally works for all statements in Python. The same is not at all true with the C equivalent: for (pch=mystring; *pch; ++pch) process(*pch) No matter how you try to pronounce the punctuation, the result is not English, it's gibberish. C is not meant to be readable as English (or as "executable pseudocode", if you prefer). Now try this with switch: switch mystring case "spam": process(thingy) That is not even remotely interpretable as an English sentence. That's not a problem for C, but it is for Python. >> Bash doesn't have separate "case" and "elcase" cases. After one case is done, the rest are skipped, just as in most other languages. > > But it has ;& and ;;& similarly to break and continue of C, that it's equivalent to case and elcase of my proposal. First, break and continue are not even remotely equivalent of your case and elcase. A break means you skip over all subsequent cases; no break means you fall through to the next case; continue is a syntax error (unless of course the switch is inside a loop, in which case it's the loop that's continued); and there is no way short of abusing goto to skip over some cases but not others. Second, bash does not have either ";;&" or ";&". It requires every case clause to end in ";;", which always skips over the rest of the cases. A ";" just separates statements within a clause. In some versions of bash, ";&" is interpreted basically the same as "&"--that is, it both ends a statement and makes a background job out of it--while in others (including all recent versions) it's a syntax error, as is ";;&". There is no way to either fall through to the next clause or skip some clauses but not others. >> I don't see how skipping over any elcase but falling through to the next case is in any way simpler than C. > > Well, because it's coherent with if-elif. See my last example in my first message. No it isn't. An elif clause is embedded in an if statement; there is no if clause you can embed on an if statement, just an entirely new and unrelated if statement. >> Well, then at least look at the limited form of pattern matching Python has in the for and assignment statements and parameter matching, and maybe look at how pattern matching is used with case statements in other languages; don't try to suggest language designs based on guesses. > > Excuse me? I know list comprehensions, lambdas and argument unpacking. Who asked you about list comprehensions or lambdas? What do you think they have to do with anything? Can you give an example that shows how they're relevant? > And I do not think you can see what I do before I post a message. No, all I can see is what you say. But you said that you don't know much about pattern matching, so I think it's fair to assume you don't know much about pattern matching. And I think that also makes it fair to assume you didn't try to learn about it, because the only alternative is that you tried and were incapable, which I think would be very insulting, and I prefer not to assume that of people. > About pattern matching in the for statement, I really don't know what they are. OK, first let's look at something you can do with a case statement in a language like ML or Haskell, translated into more Python-like syntax: case spam: of (x, y): process(x, y) This case will match only if spam is a tuple of exactly two values, and will bind x and y to those values. Obviously that syntax conflicts with your syntax for matching two distinct cases with the same clause, but ignore that for a moment. If there were suitable syntax for both, would you want that in Python? Compare this perfectly valid Python code: if value == x, y: process(x, y) x, y = value process(x, y) for x, y in pairs_of_values: process(x, y) Here, each element in pairs_of_values has to be an iterable (not necessarily a tuple) of exactly two values, and x and y are bound to the two values. That's very close to what's happening in the ML-style case statement (and the difference--the fact that it takes any iterable instead of a tuple--is probably what you'd expect from Python duck typing vs. ML static typing). Meanwhile, ML and friends let you go farther, matching by partial values: of (True, y): process(y) of Eggs(z): fry(z) The first matches only if it's a tuple of exactly two values and the first is equal to True, binding y to the second; the second matches only if it's equal to an Eggs instance constructed with exactly one argument, and binds z to that argument. Clearly not all of this fits into Python. (The last example isn't even conceptually possible, given the way object initialization works.) But it's worth considering what does and what doesn't fit rather than trying to design a feature without even knowing what the options are. >> ... and? Are you suggesting that if the switch expression is a string and the case expression a compiled regex you could automatically call match instead of testing for equality? If not, how is having regexp even relevant here? And how are recursive functions relevant? > > I'm suggesting to use if-elif with re module, if you want to use regular expression, and to use recursive functions if you want... recursive functions. I have no idea why you think recursive functions are relevant to anything being discussed here. Maybe if you can give and example of what you mean? > To be more clear, IMHO switch-case is useful if it's simple. > > >> A generator expression is equal to anything except itself, and doesn't contain anything. > > You can convert it to an iterable. Probably an overkill, but you can do it. A generator expression is already an iterable; no conversion is necessary. But your proposal was to treat tuples specially and match all other iterables as single values, which means that a generator expression would be matched as a single value, meaning it would only match itself. And again, I don't understand what the relevance is supposed to be. >> I don't know what you mean by "symbolic pattern" here. > > For what I know (not too much), in Mathematica pattern matching can be used for symbols, and symbols can be used as identifiers: > https://reference.wolfram.com/mathematica/guide/Patterns.html You're mixing up different terms here. Symbolic patterns in Mathematica match symbolic structures, not identifiers. The idea is to provide ML-style structural pattern matching, in a way that looks like intuitive "fill-in-the-blanks" in the simplest cases, but is then extended in a way that's similar to regular expressions, but more verbose and readable whenever regexps become too obtuse. Mathematica uses these patterns in all kinds of places beyond what Python and ML do, and it's a pretty cool feature, but I think you want to look at some examples instead of trying to get the idea from the formal reference docs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Apr 19 07:11:16 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 19 Apr 2014 14:11:16 +0900 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> Message-ID: <8761m5sy2z.fsf@uwakimon.sk.tsukuba.ac.jp> Skip Montanaro writes: > In other languages, the semantics of the switch statement allow the > compiler to generate more efficient code. Instead of testing each > branch of the if statement in succession looking for a match, you > evaluate the switch expression once, then use it as an index into a > jump table of some sort which points directly to the matching case > label. Sure, but Python already has such a jump table: a function-valued hash. Don't-even-think-of-mentioning-Ruby-blocks-now-ly y'rs, From bruce at leapyear.org Sat Apr 19 08:52:59 2014 From: bruce at leapyear.org (Bruce Leban) Date: Fri, 18 Apr 2014 23:52:59 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <22741-1397762051-908018@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> Message-ID: Here's a simple class that provides case like syntax. How would modifying the language be better than this? This code makes it clear that all the checks are intended to be against the same value. It can easily be extended to have structural matching or regex matching. I think case/elcase would be confusing as it would be unique to python and it's backwards -- the elcase keywords is used for the normal way that people use case statements while the case keyword implements the broken code that inexperienced C programmers frequently write. With this class, you use if/elif/else exactly the way you usually do, and notice that there is no ambiguity when checking multiple values. class Case(object): def __init__(self, value): self.value = value def __enter__(self): return self def __exit__(self, exc_type, exc_value, traceback): pass def __call__(self, *values): return self.value in values def __eq__(self, value): return self.value == value def __lt__(self, value): return self.value < value def __gt__(self, value): return self.value > value # etc. # can easily add structural matching or regex matching as well for i in range(5): print(i, end=' => ') with Case(i) as case: if case(1): print('one') elif case((2,3)): print('tuple(two, three)') elif case(2, 3): print('two or three') elif case > 3: print('more than three') else: print('unmatched') produces 0 => unmatched 1 => one 2 => two or three 3 => two or three 4 => more than three Is it worth using a context manager for this? Maybe not. But if not, I think it's even less worthwhile to modify the language to add a case statement. --- Bruce Learn how hackers think: http://j.mp/gruyere-security https://www.linkedin.com/in/bruceleban -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at gmail.com Sat Apr 19 11:56:14 2014 From: fuzzyman at gmail.com (Michael Foord) Date: Sat, 19 Apr 2014 10:56:14 +0100 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> Message-ID: On 18 April 2014 16:22, Skip Montanaro wrote: > On Fri, Apr 18, 2014 at 10:03 AM, Joao S. O. Bueno > wrote: > > It may be just me, but I fail - in a complete manner - to see how this > > syntax can offer any > > improvement on current if/elif chains. > > I haven't been following this thread closely, so I don't know if Lucas > has mentioned more than syntax, however... > > In other languages, the semantics of the switch statement allow the > compiler to generate more efficient code. Instead of testing each > branch of the if statement in succession looking for a match, you > evaluate the switch expression once, then use it as an index into a > jump table of some sort which points directly to the matching case > label. If nothing matches, you jump to the default branch (if one was > defined) or just jump over the entire switch statement (if not). > > In which case you compile in the switch values at "compile time", so the semantics are "surprising" (if you switch on a name instead of a fixed value then changing the object the name is bound to doesn't change the behaviour of the switch). So you either get fast but surprising (unpythonic) or dynamic and no different to an if/elif chain. If you restrict switch to hashable literals then you can make it fast, and nicer syntactically than the standard pattern - building a dictionary of functions. But that's quite a restriction. (You basically note this in your follow up email - but this has been the problem with previous discussions of switch/case for Python.) Michael > Skip > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From aj at erisian.com.au Sat Apr 19 13:12:47 2014 From: aj at erisian.com.au (Anthony Towns) Date: Sat, 19 Apr 2014 21:12:47 +1000 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20140419031221.GI28400@ando> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> Message-ID: On 19 April 2014 13:12, Steven D'Aprano wrote: > One motive I'd like to see is, could a switch/case statement be used to > automate table lookups? Two problems -- albeit mild ones -- with the > table lookup idiom is that the table lives longer than needed, and it > puts the table in the wrong place. This motivated Nick to suggest a > "where" block: > > card = table[tarot] where: > table = {0: "Fool", 1: "Alan Moore", 2: "High Priestess", ...} > > > solving both problems at once: the table is declared only when needed, > not before hand, and does not exist outside of the block. A third > problem, not solved by Nick's "where", is that the table requires every > case's value ahead of time, whether it will be needed or not. When the > values are constants, that's not a big deal, but they might be expensive > expressions. A related problem is that a table lookup can only hold expressions, not statement blocks. If you want statement blocks you'd have to define functions first, then reference back to them in the table, then lookup the table. (And if you just want late/lazy evaluation but not blocks, you end up having lambda expressions in your tables) It might be reasonable to consider a "switch" statement to solve just those problems; ie: case X in: 1: print("Hello, world") 2: print("Goodbye, world") 3: print("Whatever") ... else: raise Exception("unexpected value for X") would be equivalent to: __cases = {} def _(): print("Hello, world") __cases[1] = _ ... def _(): raise Exception("unexpected value for X") __cases.default = _ __cases.get(X, __cases.default)() If you treated it literally that way, building a dict up from nothing every time the "case" block was hit, it would just be syntactic sugar for if/elif, which doesn't seem valuable. (It'd save you repeating the "X ==" part of every condition, and you might get an error if you had "1:" as two different cases, which could be useful) if you had the __selector dict populated at parse time, you'd be able to reuse it on each iteration/call to the function and could get a performance win (you'd only evaluate the right hand side of the expressions once, and you'd just be doing a hash lookup on each iteration/invocation). The downside is that dynamic values in the cases would then be confusing: y = 0 def foo(x): case x in: y: return "matched y" else: return "didn't match y" foo(0) # matched y y = 1 foo(1) # didn't match y or: for i in range(10): case i in: i: print("i == i") else: print("i != i") might only print i == i when i is 0 (or might give a syntax error that i is undefined when first compiling the select block). (The above would be Alternative 4, with case instead of switch as the keyword, Option 3 from PEP 3103 I think) C/C++ doesn't have that problem because it can just issue an error at compile time that "i" isn't a constant expression. I don't think you could do anything similar in python, because you couldn't distinguish between symbolic constants and variables... You could argue that it's similar to when you have mutable default arguments to functions and expect people to just deal with it, but that doesn't seem ideal either... You could always just refactor your "switch" statement out of your function body and into a separate definition, in which case you could just define a class: def card_id(id): def dec(f): f = staticmethod(f) TarotActor_card_actions[id] = f return f return dec TarotActor_card_actions = {} class TarotActor: @classmethod def dispatch(cls, card_id): return TarotActor_card_actions[id]() @card_id(0) def Fool(): ... @card_id(1) def Magician(): ... and then just invoke TarotActor.dispatch(tarot_card) from wherever. Having to do two separate lines for the code block introduction (decorator and def, versus just a case statement) is irritating I suppose, but it's not that bad. And since it's three different levels of indentation (switch statement, cases, and code blocks), moving it to module level is arguably a good idea just for indentation's sake... Cheers, aj -- Anthony Towns From aj at erisian.com.au Sat Apr 19 14:16:36 2014 From: aj at erisian.com.au (Anthony Towns) Date: Sat, 19 Apr 2014 22:16:36 +1000 Subject: [Python-ideas] from __history__ import ... Message-ID: Antoine Pitrou wrote (on python-dev): > On Fri, 18 Apr 2014 22:31:29 -0400 > Nick Coghlan wrote: > > After spending some time talking to the folks at the PyCon Twisted > > sprints, they persuaded me that adding back the iterkeys/values/items > > methods for mapping objects would be a nice way to eliminate a key > > porting hassle for them (and likely others), without significantly > > increasing the complexity of Python 3. > I'm -1 on this. This is destroying the simplification effort of the > dict API in Python 3. I'm one of the masses who basically ignores py3 in favour of py2.7 (or 2.6 or even 2.4...) because the code I write has to run on old python. And even if I /want/ to be compatible with py3, the only way I can do that is by doing things that perform poorly in the version of python I'll actually be using in production, or by having ugly special cases and boiler plate all over the place. Things like this really are a barrier to adoption for py3... There's already a "from __future__ ..." mechanism to help with backwards incompatible changes (creating new keywords and syntax in particular). But that only helps people stay on old versions of python. So why not create the opposite: "from __history__ import ..." that allows you to use obsoleted ways of programming with new versions of python? If you want to use iter* on dicts in py3.5, require a "from __history__ import dict_iter" statement at the top of the file. For py2.7, there is no __history__ module, but it's easy to add one to your project that just says "dict_iter = True" to let the import statement succeed. In general (looking forward to py4?) you could (at least in theory) have "from __history__ import FOO" be a noop if FOO is unknown, add support for whatever historical practice if it's known and a barrier to adoption, and have it be a syntax error when it's finally no longer supported at all. That way you only need to actually define a symbol via the __history__ module when you're actually removing an obsolete idiom from the language. Cheers, aj -- Anthony Towns From ncoghlan at gmail.com Sat Apr 19 17:07:56 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Apr 2014 11:07:56 -0400 Subject: [Python-ideas] from __history__ import ... In-Reply-To: References: Message-ID: On 19 Apr 2014 08:18, "Anthony Towns" wrote: > > There's already a "from __future__ ..." mechanism to help with > backwards incompatible changes (creating new keywords and syntax in > particular). But that only helps people stay on old versions of > python. > > So why not create the opposite: "from __history__ import ..." that > allows you to use obsoleted ways of programming with new versions of > python? If you want to use iter* on dicts in py3.5, require a "from > __history__ import dict_iter" statement at the top of the file. There are two reasons for that, one technical, one more people oriented. The technical reason is that *data type* behaviours can't be isolated to a single module - calling functions and returning values means they escape easily to other code and need to behave as that code expects. This is a big reason why most of the thorny migration problems revolve around text, binary data and mappings - the *object* interfaces changed, so its hard to do module-at-a-time updates. The other reason goes back to the fact that part of the rationale for deprecating and removing legacy behaviours is so *new* users never need to learn them. The deprecation process is designed to strongly encourage incremental modernisation over time, minimising the amount of legacy cruft that hangs around indefinitely in Python code bases. Python 3 originally broke with that tradition of incremental modernisation, but the rise of code that runs in the common subset of Python 2 & 3 means that a fair bit of work is now going into enabling that model for the Python 3 transition as well. So when we add back a Python 2 feature to Python 3, it needs to be justified as either being an improvement in its own right (e.g. binary interpolation) or as being sufficiently easy to explain to new users that it doesn't significantly hurt the readability and maintainability of Python 3 code (e.g. explicit Unicode literals, my proposal to restore the mapping iter methods). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From 7vsfeu4pxg at snkmail.com Sat Apr 19 19:42:54 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Sat, 19 Apr 2014 19:42:54 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20140419031221.GI28400@ando> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> Message-ID: <20199-1397929416-249142@sneakemail.com> On 19 April 2014 05:12, Steven D'Aprano steve-at-pearwood.info | python-ideas-at-python.org| <6hqclkasyt at sneakemail.com> wrote: > In the case of your tarot > example, it is *not* a violation of DRY, because the various elif lines > are not *sources* of knowledge which may contradict each other. The > worst that will happen is that if you change the variable name "tarot" > to something else, your code will fail with a name error. You could also use another existing variable. Anyway if your > motive in introducing a switch/case statement is to DRY, I think you > need a better motive. > I completely agree. I think the main good reason to introduce a switch statement is that is more simple to use, when it can be used. > @Skip Montanaro: yes, switch statement is used in C also for code > > optimization. Frankly, I think this aspect is unimportant for CPython in > > the present time. > > Then I think we're not going to agree on the use or need for switch. > To be clear: I'm not saying you can't improve the performance of a switch statement, but IMHO currently Python has worse speed problems. On 19 April 2014 07:11, Stephen J. Turnbull stephen-at-xemacs.org | python-ideas-at-python.org| <85q573xayt at sneakemail.com> wrote: > Skip Montanaro writes: > > > In other languages, the semantics of the switch statement allow the > > compiler to generate more efficient code. Instead of testing each > > branch of the if statement in succession looking for a match, you > > evaluate the switch expression once, then use it as an index into a > > jump table of some sort which points directly to the matching case > > label. > > Sure, but Python already has such a jump table: a function-valued > hash. > Yes, I think it's more befitting than a switch. On 19 April 2014 08:52, Bruce Leban bruce-at-leapyear.org | python-ideas-at-python.org| wrote: > Here's a simple class that provides case like syntax. How would modifying > the language be better than this? > I don't like the additional unneeded Case indentation. That's why I'm proposing a syntax with switch on the same line with the first case. I think case/elcase would be confusing as it would be unique to python and > it's backwards -- the elcase keywords is used for the normal way that > people use case statements while the case keyword implements the broken > code that inexperienced C programmers frequently write. This is the point. When I was a beginner I have not tried to code Python directly, since it was an "exotic" language for me; so I read tutorials and docs. In tutorial there will be an example with elcase, and if I have C background I'll think "what damn is elcase"? And I'll read docs more carefully. And what if I read the code? If I don't know how switch statement works in Python and I read a Python code with elcase, the result will be the same. If I read an example without elcase, since it will work as a C switch statement, there's no possibility of misunderstanding. The problem with C switch is that you have to write break; at the end of each case suite, and typically you forget to do it. With my proposal you can't forget it if you know the syntax. -------------- next part -------------- An HTML attachment was scrubbed... URL: From 7vsfeu4pxg at snkmail.com Sat Apr 19 19:54:50 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Sat, 19 Apr 2014 19:54:50 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> Message-ID: <20935-1397930131-420045@sneakemail.com> On 19 April 2014 06:54, Andrew Barnert abarnert-at-yahoo.com | python-ideas-at-python.org| <3en9kh2cbt at sneakemail.com> wrote: > switch mystring case "spam": > process(thingy) > > That is not even remotely interpretable as an English sentence. That's not > a problem for C, but it is for Python. > I'll think about a possible solution. I don't see how skipping over any elcase but falling through to the next >>> case is in any way simpler than C. >> >> Well, because it's coherent with if-elif. See my last example in my >> first message. >> > No it isn't. An elif clause is embedded in an if statement; there is no if > clause you can embed on an if statement, just an entirely new and unrelated > if statement. > I can assert, without fear of contradictions, that my proposed syntax is the closest to the if-elif syntax of all the switch statement proposed until now in the age of Python. About off-topic arguments: bash does not have either ";;&" or ";&" > http://www.gnu.org/software/bash/manual/html_node/Conditional-Constructs.html Who asked you about list comprehensions or lambdas? What do you think they > have to do with anything? > [...] > I have no idea why you think recursive functions are relevant to anything > being discussed here. > [...]your proposal was to treat tuples specially and match all other > iterables as single values, which means that a generator expression would > be matched as a single value, meaning it would only match itself. And > again, I don't understand what the relevance is supposed to be. I premise that pattern recognition is outside the scope of my switch proposal. Anyway: 1. About list comprehensions and lambdas, you talked about assignment statements, and AFAIK the only pattern matching things that are somewhat related to assignment statement are the possibility to filter a list comprehension and the use of lambdas. If you mean something different you should be more explicit, instead of saying "what? where? why?" 2. About recursive function, I wronged. Sorry but it was late. 3. About generators, you can create a generator, convert it to an iterable and unpack it in a case_expr. I could also extend the current syntax and support a "generator unpacking". This way you'll have a limited alternative to pattern matching in some cases in a switch statement without the need to create a static iterable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-ideas at hmmz.org Sun Apr 20 00:51:10 2014 From: dw+python-ideas at hmmz.org (David Wilson) Date: Sat, 19 Apr 2014 23:51:10 +0100 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> Message-ID: <96949f3566ab45335852c172c7bdf970@hmmz.org> Hi Donald! Thanks for replying, > And honestly with the new PR I?ve laid out in Travis adding any older > versions to be supported out of the box is simple and the biggest > barrier is likely convincing the Travis folks they are needed (For > example, if you?re distributing on PyPI the number of 2.5 or 3.1 users > are *tiny* https://caremad.io/blog/a-look-at-pypi-downloads/). > > As far as I?m aware the Travis-CI folks are working on Windows support > but there is currently no ETA on it as they have to figure out > licensing concerns and the like. Travis can work well, but the effort involved in maintaining it has, at least for me, been unjustifiably high: in total I've spent close to a man-week in the past 6 months trying various configurations, and fixing things up once a month, before resorting to what I have now. In all, I have probably spent 10x the time maintaining Travis as I have actually writing comprehensive tests. It cannot reasonably be expected that project maintainers should pay similarly, or ideally even need to be aware their package is undergoing testing someplace. My biggest gripe with Travis, though, is that they can and do remove things that break what for me seems trivial functionality. This is simply the nature of a one-size-fits-all service. From what I gather, they previously removed old Python releases from their base image to reduce its size. Windows licensing is hard, but given the scope of either Travis or the PSF, I'd be surprised if there wasn't some person or group at Microsoft to solve it for us, especially considering the new climate under Nadella. I know at least that they offer free Azure time to open source projects, even something like this could be used. > The sanest way of doing this is to manage a pool of VM workers, pull > one off the pool, run tasks on it, and then destroy it and let the > pool spawn a new one. This puts a bottleneck on how many tests you can > run both in how large your pool is, and in how fast your VMs boot. > Look at a project like https://travis-ci.org/pyca/cryptography which > builds 40 different configurations for each ?build? each taking about > 8 minutes each. They are curious numbers, perhaps indicative of overcontention on extra-whimpy cloud VMs. On uncontended spinning rust with 2GB RAM my measily Core 2 Duo gets: Clone: 4.7s dev_requirements.txt + test vectors + pip install -e: 24.6s py.test: 2m10s For a package on the opposite end of the spectrum, in terms of test comprehensiveness and size, py-lmdb: Clone: 0.99s pip install -e: 2.646s py.test: 0.996s > So if you have a pool of 5 VMs, you can run 5 tests at a time, so that > splits that 40 into 5 chunks, and you get roughly 64 minutes to > process 40 builds, plus we?ll say our machines boot in roughly 60s (a > little fast for cloud servers, but not unreasonable), It's easy to beat 60 seconds for a farm: resuming a pre-booted 2GB VM from cached snapshot with Qemu/KVM takes between 1.5s (465mb dirty) to 3s (2gb dirty), again on a whimpy Core 2 Duo. Actually it's surprising the numbers are so high, Qemu's IO code seems not the most efficient. Dirty numbers are interesting as tooling can be warmed into the VM page cache prior to snapshot. Since Python has solid side-by-side support for over 10 years, only one per-OS base image need exist with all versions installed and cache-warm prior to test (about 1GB total if you include standard libraries). Just to be clear, this is entirely free in the cost of resuming a snapshot. Further assuming local mirrors of PyPI and source repositories, it's easy to see how a specialized farm can vastly improve efficiency compared to a general solution. > so there?s an additional 4-5 minutes just in booting. So roughly an > hour and 10 minutes for a single test run if with 5 VMs for just a > single project. (In reality Cryptography has more than 40 builds > because it uses Travis and jenkins together to handle things travis > doesn?t). > So the machine cost is high, you?re probably looking at let?s just say > a worker pool of 20 (though I think to actually replace Travis CI it?d > need to be much higher) 40*2m40s works out to about 1h45m whimpy CPU-hours for a complete cycle, but this assumes results of all 40 workers are needed to gauge project health. In reality there are perhaps 3-6 configurations needing priority for prompt feedback (say CPyMac, CPyLinux, CPyWin, PyPyMac, PyPyLinux, PyPyWin). Ideally py-lmdb has about 32 configurations to test, including Windows. Using earlier numbers that's (32*7.5s) or about 7m whimpy CPU hours, however 45 seconds is sufficient to test the main configurations. It doesn't matter if Python2.5 breaks and it's not obvious for 24 hours, since in this case, releases are only made at most once per month. Assuming the average PyPI package lies somewhere between Cryptography and py-lmdb, and there are about 42k packages, roughly averaging this out gives: (7.5s*32 + 2m40s*40)/(32+40) = 1m32s, or about 1088 whimpy CPU hours to completely run one configuration of each package in PyPI. Now instead of whimpy CPUs we have an 8-core Xeon, pessimistically assuming the Xeon is only 20% faster, that gives 108.88 beefy 8-core-CPU hours to rebuild PyPI once. Assuming in a day 3% of packages are updated (which seems high), that's 3h15m 8-core-CPU hours to test each updated package from a fresh checkout in a fresh VM against a single Python version, which as we've seen, is the worst case behaviour since side-by-side versions are possible. In summary, one 8-core machine might suffice (allowing for napkin math) to retest 3% of PyPI in one config at least 7 times a day, or assuming each package has 6 primary configurations, one "all important configs" cycle once per day. > of roughly 4GB machines (last I recall travis machines were 3gig and > change) which comes out to roughly 1k to 2k in server costs just for > that pool. Add into that whatever support machines you would need > (queues, web nodes, database servers, etc) you?re probably looking in > the 2-4k range just for the servers once all was said and done. If one build per 4 hours sufficed for most projects, $1k/month seems like a good cap: a VM with comparable specs to the above scenario, GCE's n1-standard-8, costs around $275/month to run 24/7, assuming the project couldn't find a sponsor of multiple machines, which I suspect would be quite easy. The above estimates are a little optimistic: in addition to 2-4GB guest RAM per core, the host would need at least 8-32GB more to keep hot parts of the base image filesystems cached to achieve the time estimates. However, I've never seen any Python extension needing 4GB to build. Regarding supplementary services, a farm produces logs and perhaps assets for which a static file bucket suffices, and consumes jobs from a queue, which wouldn't require more beef than an SQLite database, and I'd be impressed if 40 jobs * 45k packages would fill 2GB. > I believe the code cost would also be fairly high. There isn?t really > an off the shelf solution that is going to work for this. .. There is > a good chance some things could be reused from Openstack but I?m > fairly sure some of it is specific enough to Openstack that it?d still > require a decent amount of coding to make it generic enough to work. It's easy to overdesign and overspec these things, but the code and infrastructure involved is fairly minimal, especially to produce something basic that just ensures, e.g. recently published or manually triggered-via-webhook packages get queued and tested. The most complex aspect is likely maintaining reproducable base images, versioning them and preventing breakage for users during updates, but that is almost a planning problem rather than an implementation problem. On saying this, though, I can imagine a useful versioning policy as simple as two paragraphs, and an image/snapshot builder as simple as two shell scripts. Integrating complex "best practices" off-the-shelf components is an example of where simple projects often explode.. Qemu is literally a self contained command-line tool, no more complex to manage the execution of than, say, wget. With a suitably prepped snapshot, all required is to run qemu, allow it 10 mins to complete, and read job results via a redirected port. > This means that it?s likely not going to be something we can just set > it up and forget about it and will require an active infrastructure > team to handle it. Now Python happens to have one of those, but we?re > mostly volunteers with part times for on call stuff (who also > volunteer for other stuff too!) and this would be a significant > increase I believe in that work load. That's very true, though looking at the lack of straightforward Python solution at the ecosystem level, it also seems quite a small cost. > It?s my personal opinion that a sort of ?test during development? CI > system like Travis is not something that making Python specific is > very useful in the long run. Travis is open source and they are > incredibly open to things that will make Python a more first class > citizen there (disclaimer: I?m friends with them and I?ve helped add > the Python support to start with, and then improve it over time). One > obvious flaw in this is that Travis only supports github while there > are others on Bitbucket or even their own hosting, and for those > people this idea might be much more attractive. > > I do think there is maybe room for a ?release testing? system and I > definitely think there is room for a build farm (Which is a much > smaller scale since the whole of PyPI pushes releases far less often > than push commits to branches or make PRs or what have you). It's not clear what of scope would work best for a Python-specific system. For my needs, I'm only interested in removing the effort required to ensure my packages get good, reasonably timely coverage, and not worry about things like Windows licenses, or my build exploding monthly due to base image changes sacrificing Python-related functionality for something else. The size and complexity of the service could creep up massively, especially if attempting to compete with e.g. Travis' 5-minutes-or-less turnaround time, but at least for me, those fast turnaround times aren't particularly useful, I just need something low effort that works. Regarding the value of a Python-specific system, almost identical infrastructure could be used for your Wheel farm, or running any kind of "ecosystem lint", like detecting setup.py /etc/passwd writes and suchlike. Granted though, these could be also be solved independently. Finally it's not obvious to me that this is absolutely a good idea, however I'm willing to argue for it for as long as I'm procrastinating on setting up my own Jenkins install :-P David From ncoghlan at gmail.com Sun Apr 20 01:56:36 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Apr 2014 19:56:36 -0400 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <96949f3566ab45335852c172c7bdf970@hmmz.org> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> <96949f3566ab45335852c172c7bdf970@hmmz.org> Message-ID: On 19 Apr 2014 18:52, "David Wilson" wrote: > > Finally it's not obvious to me that this is absolutely a good idea, however I'm willing to argue for it for as long as I'm procrastinating on setting up my own Jenkins install :-P I actually think it's a good idea, I'd just like to approach the idea indirectly by building out CPython's test infrastructure further first, before we try to boil the ocean that is PyPI :) Cheers, Nick. > > > > David > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Sun Apr 20 03:34:53 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 19 Apr 2014 21:34:53 -0400 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <96949f3566ab45335852c172c7bdf970@hmmz.org> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> <96949f3566ab45335852c172c7bdf970@hmmz.org> Message-ID: <8FF25312-584E-4C52-939B-526C2473E2F4@stufft.io> On Apr 19, 2014, at 6:51 PM, David Wilson wrote: > Hi Donald! > > Thanks for replying, > > >> And honestly with the new PR I?ve laid out in Travis adding any older >> versions to be supported out of the box is simple and the biggest >> barrier is likely convincing the Travis folks they are needed (For >> example, if you?re distributing on PyPI the number of 2.5 or 3.1 users >> are *tiny* https://caremad.io/blog/a-look-at-pypi-downloads/). >> As far as I?m aware the Travis-CI folks are working on Windows support >> but there is currently no ETA on it as they have to figure out >> licensing concerns and the like. > > Travis can work well, but the effort involved in maintaining it has, at least for me, been unjustifiably high: in total I've spent close to a man-week in the past 6 months trying various configurations, and fixing things up once a month, before resorting to what I have now. In all, I have probably spent 10x the time maintaining Travis as I have actually writing comprehensive tests. > > It cannot reasonably be expected that project maintainers should pay similarly, or ideally even need to be aware their package is undergoing testing someplace. > > My biggest gripe with Travis, though, is that they can and do remove things that break what for me seems trivial functionality. This is simply the nature of a one-size-fits-all service. From what I gather, they previously removed old Python releases from their base image to reduce its size. I?ve not had anywhere near that type of experience trying to get anything setup on Travis. The one exception had been when I?ve attempted to use their OSX builders, however a big portion of that is because their travis builders do not have Python configured by default. That?s not to say you didn?t have that behavior, but generally the worst I?ve had to do is install a python other than what they?ve provided, which has been trivially easy to do as well (See: https://github.com/pypa/pip/blob/develop/.travis.yml#L18-L23). Looking over the history of the pip .travis.yml the only times it?s really changes were to add versions or because we were trying out some new thing (Like using pytest-xdist). For the record They?ve never supported 2.4, and I recommended they drop 2.5 and 3.1 because the incoming traffic on PyPI don?t really justify keeping it in the base image because a larger base image takes a longer period of time to boot which slows down the entire build queue. For projects that want to keep it, it?s trivial to add back in a way similar to how pip has added 3.4 support. > > Windows licensing is hard, but given the scope of either Travis or the PSF, I'd be surprised if there wasn't some person or group at Microsoft to solve it for us, especially considering the new climate under Nadella. I know at least that they offer free Azure time to open source projects, even something like this could be used. Sure, Licensing is a solvable problem, but it?s still a problem :) There are other problems too of course, such as handling fundamental differences in how the two platforms interact (Easy to SSH into a *nix box, not so much a Windows box). > > >> The sanest way of doing this is to manage a pool of VM workers, pull >> one off the pool, run tasks on it, and then destroy it and let the >> pool spawn a new one. This puts a bottleneck on how many tests you can >> run both in how large your pool is, and in how fast your VMs boot. > >> Look at a project like https://travis-ci.org/pyca/cryptography which >> builds 40 different configurations for each ?build? each taking about >> 8 minutes each. > > They are curious numbers, perhaps indicative of overcontention on extra-whimpy cloud VMs. On uncontended spinning rust with 2GB RAM my measily Core 2 Duo gets: > > Clone: 4.7s > dev_requirements.txt + test vectors + pip install -e: 24.6s > py.test: 2m10s > > For a package on the opposite end of the spectrum, in terms of test comprehensiveness and size, py-lmdb: > > Clone: 0.99s > pip install -e: 2.646s > py.test: 0.996s I believe that Travis is using OpenVM VMs which may or may not be on the greatest hardware. A different VM provider would probably give better performance. > > >> So if you have a pool of 5 VMs, you can run 5 tests at a time, so that >> splits that 40 into 5 chunks, and you get roughly 64 minutes to >> process 40 builds, plus we?ll say our machines boot in roughly 60s (a >> little fast for cloud servers, but not unreasonable), > > It's easy to beat 60 seconds for a farm: resuming a pre-booted 2GB VM from cached snapshot with Qemu/KVM takes between 1.5s (465mb dirty) to 3s (2gb dirty), again on a whimpy Core 2 Duo. Actually it's surprising the numbers are so high, Qemu's IO code seems not the most efficient. > > Dirty numbers are interesting as tooling can be warmed into the VM page cache prior to snapshot. Since Python has solid side-by-side support for over 10 years, only one per-OS base image need exist with all versions installed and cache-warm prior to test (about 1GB total if you include standard libraries). Just to be clear, this is entirely free in the cost of resuming a snapshot. All of those things require even more ongoing support since we?d have to support the host machines as well and not just reuse some providers VM/cloud images. > > Further assuming local mirrors of PyPI and source repositories, it's easy to see how a specialized farm can vastly improve efficiency compared to a general solution. As far as I?m aware Travis has been willing to setup a PyPI mirror, it?s mostly been nobody has helped them set one up :) > > >> so there?s an additional 4-5 minutes just in booting. So roughly an >> hour and 10 minutes for a single test run if with 5 VMs for just a >> single project. (In reality Cryptography has more than 40 builds >> because it uses Travis and jenkins together to handle things travis >> doesn?t). > >> So the machine cost is high, you?re probably looking at let?s just say >> a worker pool of 20 (though I think to actually replace Travis CI it?d >> need to be much higher) > > 40*2m40s works out to about 1h45m whimpy CPU-hours for a complete cycle, but this assumes results of all 40 workers are needed to gauge project health. In reality there are perhaps 3-6 configurations needing priority for prompt feedback (say CPyMac, CPyLinux, CPyWin, PyPyMac, PyPyLinux, PyPyWin). > > Ideally py-lmdb has about 32 configurations to test, including Windows. Using earlier numbers that's (32*7.5s) or about 7m whimpy CPU hours, however 45 seconds is sufficient to test the main configurations. It doesn't matter if Python2.5 breaks and it's not obvious for 24 hours, since in this case, releases are only made at most once per month. > > Assuming the average PyPI package lies somewhere between Cryptography and py-lmdb, and there are about 42k packages, roughly averaging this out gives: (7.5s*32 + 2m40s*40)/(32+40) = 1m32s, or about 1088 whimpy CPU hours to completely run one configuration of each package in PyPI. > > Now instead of whimpy CPUs we have an 8-core Xeon, pessimistically assuming the Xeon is only 20% faster, that gives 108.88 beefy 8-core-CPU hours to rebuild PyPI once. > > Assuming in a day 3% of packages are updated (which seems high), that's 3h15m 8-core-CPU hours to test each updated package from a fresh checkout in a fresh VM against a single Python version, which as we've seen, is the worst case behaviour since side-by-side versions are possible. > > In summary, one 8-core machine might suffice (allowing for napkin math) to retest 3% of PyPI in one config at least 7 times a day, or assuming each package has 6 primary configurations, one "all important configs" cycle once per day. > Going back to cryptography, those 40 test runs are *just* the ones that can run on Travis. There is also an additional 54 runs for Windows, OSX, FreeBSD, and other Linux installs. The cryptography project runs roughly 15 of those builds almost every day, and goes upwards of 30 builds a day on other days. So that?s 1410-2820 total runs, even with a 2m40s run you?re looking at that single project taking 60-125hours of CPU build time a day. Now this project is a fairly intensive one, but it?s useful to look at worst case uses in order to determine scaling. > >> of roughly 4GB machines (last I recall travis machines were 3gig and >> change) which comes out to roughly 1k to 2k in server costs just for >> that pool. Add into that whatever support machines you would need >> (queues, web nodes, database servers, etc) you?re probably looking in >> the 2-4k range just for the servers once all was said and done. > > If one build per 4 hours sufficed for most projects, $1k/month seems like a good cap: a VM with comparable specs to the above scenario, GCE's n1-standard-8, costs around $275/month to run 24/7, assuming the project couldn't find a sponsor of multiple machines, which I suspect would be quite easy. > > The above estimates are a little optimistic: in addition to 2-4GB guest RAM per core, the host would need at least 8-32GB more to keep hot parts of the base image filesystems cached to achieve the time estimates. However, I've never seen any Python extension needing 4GB to build. > > Regarding supplementary services, a farm produces logs and perhaps assets for which a static file bucket suffices, and consumes jobs from a queue, which wouldn't require more beef than an SQLite database, and I'd be impressed if 40 jobs * 45k packages would fill 2GB. > > >> I believe the code cost would also be fairly high. There isn?t really >> an off the shelf solution that is going to work for this. .. There is >> a good chance some things could be reused from Openstack but I?m >> fairly sure some of it is specific enough to Openstack that it?d still >> require a decent amount of coding to make it generic enough to work. > > It's easy to overdesign and overspec these things, but the code and infrastructure involved is fairly minimal, especially to produce something basic that just ensures, e.g. recently published or manually triggered-via-webhook packages get queued and tested. > > The most complex aspect is likely maintaining reproducable base images, versioning them and preventing breakage for users during updates, but that is almost a planning problem rather than an implementation problem. On saying this, though, I can imagine a useful versioning policy as simple as two paragraphs, and an image/snapshot builder as simple as two shell scripts. > > Integrating complex "best practices" off-the-shelf components is an example of where simple projects often explode.. Qemu is literally a self contained command-line tool, no more complex to manage the execution of than, say, wget. With a suitably prepped snapshot, all required is to run qemu, allow it 10 mins to complete, and read job results via a redirected port. > > >> This means that it?s likely not going to be something we can just set >> it up and forget about it and will require an active infrastructure >> team to handle it. Now Python happens to have one of those, but we?re >> mostly volunteers with part times for on call stuff (who also >> volunteer for other stuff too!) and this would be a significant >> increase I believe in that work load. > > That's very true, though looking at the lack of straightforward Python solution at the ecosystem level, it also seems quite a small cost. > > >> It?s my personal opinion that a sort of ?test during development? CI >> system like Travis is not something that making Python specific is >> very useful in the long run. Travis is open source and they are >> incredibly open to things that will make Python a more first class >> citizen there (disclaimer: I?m friends with them and I?ve helped add >> the Python support to start with, and then improve it over time). One >> obvious flaw in this is that Travis only supports github while there >> are others on Bitbucket or even their own hosting, and for those >> people this idea might be much more attractive. >> I do think there is maybe room for a ?release testing? system and I >> definitely think there is room for a build farm (Which is a much >> smaller scale since the whole of PyPI pushes releases far less often >> than push commits to branches or make PRs or what have you). > > It's not clear what of scope would work best for a Python-specific system. For my needs, I'm only interested in removing the effort required to ensure my packages get good, reasonably timely coverage, and not worry about things like Windows licenses, or my build exploding monthly due to base image changes sacrificing Python-related functionality for something else. > > The size and complexity of the service could creep up massively, especially if attempting to compete with e.g. Travis' 5-minutes-or-less turnaround time, but at least for me, those fast turnaround times aren't particularly useful, I just need something low effort that works. > > Regarding the value of a Python-specific system, almost identical infrastructure could be used for your Wheel farm, or running any kind of "ecosystem lint", like detecting setup.py /etc/passwd writes and suchlike. Granted though, these could be also be solved independently. > > Finally it's not obvious to me that this is absolutely a good idea, however I'm willing to argue for it for as long as I'm procrastinating on setting up my own Jenkins install :-P > > > David > ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Sun Apr 20 03:36:05 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 19 Apr 2014 21:36:05 -0400 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> <96949f3566ab45335852c172c7bdf970@hmmz.org> Message-ID: On Apr 19, 2014, at 7:56 PM, Nick Coghlan wrote: > > On 19 Apr 2014 18:52, "David Wilson" wrote: > > > > Finally it's not obvious to me that this is absolutely a good idea, however I'm willing to argue for it for as long as I'm procrastinating on setting up my own Jenkins install :-P > > I actually think it's a good idea, I'd just like to approach the idea indirectly by building out CPython's test infrastructure further first, before we try to boil the ocean that is PyPI :) > As you can probably guess from my posts, I don?t think trying to replace Travis-CI is a good idea. What they do is *a lot* of work and maintenance and takes a lot of capacity, even if we restrict it to just Python projects. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From bruce at leapyear.org Sun Apr 20 04:17:23 2014 From: bruce at leapyear.org (Bruce Leban) Date: Sat, 19 Apr 2014 19:17:23 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20199-1397929416-249142@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> Message-ID: On Sat, Apr 19, 2014 at 10:42 AM, Lucas Malor <7vsfeu4pxg at snkmail.com>wrote: > On 19 April 2014 08:52, Bruce Leban bruce-at-leapyear.org | > python-ideas-at-python..org | < > vajg1g2cqt at sneakemail.com> wrote: > >> Here's a simple class that provides case like syntax. How would modifying >> the language be better than this? >> > > I don't like the additional unneeded Case indentation. That's why I'm > proposing a syntax with switch on the same line with the first case. > I found that the ugliest part of your proposal since it hides the first case and at the same time makes it appear special when it isn't. And merely saving indentation is not IMHO a sufficient reason to add a new feature to the language especially since you're not actually saving any. The indentation isn't required with my little Case class. The following works just as well but lacks the visible scoping that the with statement provides (of course the scope of the case variable leaks in either version, but I could modify my class to fail if invoked after __exit__ or __del__). for i in range(5): print(i, end=' => ') case = Case(i) if case(1): print('one') elif case((2,3)): print('tuple(two, three)') elif case(2, 3): print('two or three') elif case > 3: print('more than three') else: print('unmatched') --- Bruce Learn how hackers think: http://j.mp/gruyere-security https://www.linkedin.com/in/bruceleban -------------- next part -------------- An HTML attachment was scrubbed... URL: From ctb at msu.edu Sun Apr 20 06:01:22 2014 From: ctb at msu.edu (C. Titus Brown) Date: Sat, 19 Apr 2014 21:01:22 -0700 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <96949f3566ab45335852c172c7bdf970@hmmz.org> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> <96949f3566ab45335852c172c7bdf970@hmmz.org> Message-ID: <20140420040122.GT24482@idyll.org> [ munch - great conversation! ] On Sat, Apr 19, 2014 at 11:51:10PM +0100, David Wilson wrote: > My biggest gripe with Travis, though, is that they can and do remove > things that break what for me seems trivial functionality. This is > simply the nature of a one-size-fits-all service. From what I gather, > they previously removed old Python releases from their base image to > reduce its size. We've been using Jenkins connected to github on our khmer project. So far it's been rather fragile so Michael Crusoe (developer guy) is thinking about things like Travis. hmm. :) > Windows licensing is hard, but given the scope of either Travis or the > PSF, I'd be surprised if there wasn't some person or group at Microsoft > to solve it for us, especially considering the new climate under > Nadella. I know at least that they offer free Azure time to open source > projects, even something like this could be used. I am happy to broker introductions at MS if someone is committed to doing the work :) Note also that Mozilla and Numpy folk might have useful thoughts in this direction -- Mozilla does some insane number of builds per release (due to all the cell phones they need to support!) and the various Numpy folk mentioned to me that they were talking about a build farm. >> This means that it???s likely not going to be something we can just set >> it up and forget about it and will require an active infrastructure >> team to handle it. Now Python happens to have one of those, but we???re >> mostly volunteers with part times for on call stuff (who also >> volunteer for other stuff too!) and this would be a significant >> increase I believe in that work load. I and others have tried various angles on this in the past and indeed have fallen down in precisely this way... maintenance is just a bear. cheers, --titus -- C. Titus Brown, ctb at msu.edu From jbvsmo at gmail.com Sun Apr 20 06:30:08 2014 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Sun, 20 Apr 2014 01:30:08 -0300 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <22741-1397762051-908018@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> Message-ID: Python already supports switch...case statements. with switch(foo) as case: with case(10): print('foo is 10') with case.in_(15, 16, 17): print('foo might be 15 or 16 or 17') with case.default: print('Whatever...') You just need to implement the switch and case things to match the code above... Also, an if..elif..else block will be simpler to use -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Apr 20 16:04:31 2014 From: mertz at gnosis.cx (David Mertz) Date: Sun, 20 Apr 2014 07:04:31 -0700 Subject: [Python-ideas] Add a datatype collections.ListView In-Reply-To: References: Message-ID: On Apr 20, 2014 3:02 AM, "Tal Einat" wrote: > > FYI your handling of negative indexes is buggy. > It's horrible! I'm not quite sure it can even be called "handling." Take my example only as a prototype of the concept that lets me run the benchmark. That said, doing the negative indices right should just be some arithmetic on self.start and self.stop, not anything too complicated or differently performing. > > On Thu, Apr 17, 2014 at 6:49 PM, David Mertz wrote: >> >> Following Brandon Rhodes very nice PyCon 2014 talk on datatypes, I was struck by increased guilt over a program I am developing. I spoke with him in the hallway about a good approach to improving performance while keeping code nice looking. >> >> The basic idea in my program is that I have a large(ish) list of tokens that I generate in parsing a special language, and I frequently take slices from this list--and often enough, slices out of those slices. In some cases, the most straightforward approach in code is a slice-to-end, e.g.: >> >> start = find_thing(tokens) >> construct_in = tokens[start:] >> >> Obviously, this winds up doing a lot of copying (of object references, not of actual data, but still). Often the above step is followed by something like: >> >> end = find_end(construct_in) >> construct = construct_in[:end] >> >> This second step isn't bad in my case since the actual construct will be dozens of tokens, not thousands or millions, and once I find it I want to keep it around and process it further. >> >> I realize, of course, that I could program my 'find_end()' differently so that it took a signature more like 'find_end(tokens, start=start)'. But with recursion and some other things I do, this becomes inelegant. >> >> What I'd really like is a "ListView" that acts something like NumPy's non-copying slices. However numpy, of course, only deals with arrays and matrices of uniform numeric types. I want a non-copying "slice" of a list of generic Python objects. Moreover, I believe that such a type is useful enough to be worth including in the collections module generally. >> >> As an initial implementation, I created the below. In the module self-tests, the performance increase is about 100x, but the particular ad-hoc benchmark I wrote isn't necessarily well-representative of all use-cases. >> >> % python3 ListView.py >> A bunch of slices from list: 1.29 seconds >> A bunch of slices from DummyListView: 1.19 seconds >> A bunch of slices from ListView: 0.01 seconds >> ------------------------------------------------------------------- >> ### ListView.py ### >> import sys >> from time import time >> from random import randint, random >> from collections import Sequence >> >> class DummyListView(Sequence): >> def __init__(self, l): >> self.list = l >> def __len__(self): >> return len(self.list) >> def __getitem__(self, i): >> return self.list[i] >> >> class ListView(Sequence): >> def __init__(self, seq, start=0, stop=None): >> if hasattr(seq, '__getitem__'): >> self.list = seq >> else: >> self.list = list(seq) >> self.start = start >> self.stop = len(self.list) if stop is None else stop >> >> def __len__(self): >> return self.stop - self.start >> >> def __getitem__(self, i): >> if isinstance(i, slice): >> start = self.start if i.start is None else self.start+i.start >> if i.stop is None: >> stop = self.stop >> else: >> stop = self.start + i.stop >> return ListView(self.list, start, stop) >> else: >> val = self.list[i+self.start] >> if i < 0: >> return val >> elif not self.start <= i+self.start < self.stop: >> raise IndexError("View of sequence [%d:%d], index %d" % ( >> self.start, self.stop, i)) >> return val >> >> def __str__(self): >> return "ListView of %d item list, slice [%d:%d]" % ( >> len(self.list), self.start, self.stop) >> >> def __repr__(self): >> return "ListView(%s)" % self.list[self.start:self.stop] >> >> def to_list(self): >> return list(self.list[self.start:self.stop]) >> >> class Thing(object): >> def __init__(self, x): >> self.x = x >> def __repr__(self): >> return "Thing(%f)" % self.x >> >> if __name__ == '__main__': >> NUM = 100000 >> things = [Thing(random()) for _ in range(NUM)] >> slices = [sorted((randint(0, NUM-1), randint(0, NUM-1))) >> for _ in range(100)] >> offset = randint(0, 100) >> for name, cls in (("list", list), >> ("DummyListView", DummyListView), >> ("ListView",ListView)): >> begin = time() >> s = "A bunch of slices from %s: " % name >> print(s.rjust(38), end='') >> sys.stdout.flush() >> l = cls(things) >> for i in range(8): >> for start, stop in slices: >> sl = l[start:stop] >> size = stop-start >> for i in range(3): >> subslice1 = sl[:offset] >> subslice2 = sl[offset:] >> print("%0.2f seconds" % (time()-begin)) >> >> >> -- >> Keeping medicines from the bloodstreams of the sick; food >> from the bellies of the hungry; books from the hands of the >> uneducated; technology from the underdeveloped; and putting >> advocates of freedom in prisons. Intellectual property is >> to the 21st century what the slave trade was to the 16th. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Apr 20 16:08:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Apr 2014 10:08:35 -0400 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <20140420040122.GT24482@idyll.org> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> <96949f3566ab45335852c172c7bdf970@hmmz.org> <20140420040122.GT24482@idyll.org> Message-ID: On 20 Apr 2014 00:02, "C. Titus Brown" wrote: > > [ munch - great conversation! ] > > On Sat, Apr 19, 2014 at 11:51:10PM +0100, David Wilson wrote: > > My biggest gripe with Travis, though, is that they can and do remove > > things that break what for me seems trivial functionality. This is > > simply the nature of a one-size-fits-all service. From what I gather, > > they previously removed old Python releases from their base image to > > reduce its size. > > We've been using Jenkins connected to github on our khmer project. > So far it's been rather fragile so Michael Crusoe (developer guy) > is thinking about things like Travis. hmm. :) > > > Windows licensing is hard, but given the scope of either Travis or the > > PSF, I'd be surprised if there wasn't some person or group at Microsoft > > to solve it for us, especially considering the new climate under > > Nadella. I know at least that they offer free Azure time to open source > > projects, even something like this could be used. > > I am happy to broker introductions at MS if someone is committed to doing > the work :) Steve Dower (from MS) and the other PTVS folks have been doing a fair bit of work around helping out with CPython, and has been looking at ways to simplify the CPython installer build process. If you have additional points of contact within MS, that could be a good thing :) > Note also that Mozilla and Numpy folk might have useful thoughts in this > direction -- Mozilla does some insane number of builds per release (due to all > the cell phones they need to support!) and the various Numpy folk mentioned to > me that they were talking about a build farm. > > >> This means that it???s likely not going to be something we can just set > >> it up and forget about it and will require an active infrastructure > >> team to handle it. Now Python happens to have one of those, but we???re > >> mostly volunteers with part times for on call stuff (who also > >> volunteer for other stuff too!) and this would be a significant > >> increase I believe in that work load. > > I and others have tried various angles on this in the past and indeed have > fallen down in precisely this way... maintenance is just a bear. Yeah, this is part of why a key goal for me this year is getting some *paid* developer time going into the CPython workflow tools, as well as more exploration of cases where PaaS usage may lower maintenance burdens for some of our hosted services. If we can get both the CPython workflow tools and PyPI into a good state, *then* I think it may make sense to look at more ambitious things. Cheers, Nick. > > cheers, > --titus > -- > C. Titus Brown, ctb at msu.edu > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Mon Apr 21 01:12:44 2014 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 21 Apr 2014 09:12:44 +1000 Subject: [Python-ideas] 'from os.path import FILE, DIR' or internal structure of filenames In-Reply-To: References: Message-ID: <20140420231244.GA36309@cskk.homeip.net> On 20Apr2014 14:41, anatoly techtonik wrote: >On Sun, Sep 29, 2013 at 2:26 AM, Cameron Simpson wrote: >> -1 for any names commencing with __ (or even _). > >So you like it DIR, PATH As per-module/file predefined global "constants"? Possibly. Still -0, but open to argument on naming issues. >> -1 for new globals. > >So you want this: > from os.path import DIR, PATH I'm less good on these. That would import some staticish values (or are they to be functions?) which can't be right for the importing module. I suspect I'm misunderstanding your intent here. >> -1 because I can imagine wanting different nuances on the definitions >> above; in particular for DIR I can well imagine wanting bare >> dirname(abspath(FILE)) - semanticly different to your construction. >> There's lots of scope for bikeshedding here. > >Assuming that FILE is always absolute, how that: > dirname(abspath(FILE)) >is different from that: > abspath(dirname(FILE)) >? Depends what "absolute" means. When the source file was obtained by traversing a symlink, is FILE naive (ignored the symlink, just has to start at "/") or resolved (absolute path to FILE which does not traverse a symlink)? I can imagine use cases for both, and therefore the bikeshedding. >> -1 because this is trivial trival code. > >Code is trivial, but the problem is not. In particular, this code >doesn't work after os.chdir I'm fairly sure I acknowledged this. [...] Aha: On Mon, 30 Sep 2013 08:17:46 +1000 I wrote: Of course, chdir and passing paths to programs-which-are-not-my-children present scope for wanting abspath, but in the very common simple case: unnecessary and therefore undesirable. And I'm aware that modules-inside-zip-files don't work with this; let us ignore that; they won't work with abspath either:-) And what should happen for code from zipfiles? Something must happen, even if it is just NameError or the like, for "not supported". >> -1 because you can do all this with relative paths anyway, no need for abspath > >Non reliable because of os.chdir As mentioned above, yes. But very often irrelevant. (Hmm, maybe not in the general case of library code imported before a chdir and used afterwards.) Chdir's global nature (witin the process) is one reason casual use of it is discouraged; an amazing amount of stuff can be broken by a chdir. >> -1 because I can imagine being unable to compute abspath in certain >> circumstances ( certainly on older UNIX systems you could be >> inside a directory without sufficient privileges to walk back >> up the tree for getcwd and its equivalents ) > >Fear and uncertainty. Can you confirm that Python is able to launch script, but >abspath fails in these circumstances? There might be a bc break in 3.5 I'd need a suitable system today. On Linux, getcwd() has been a system call for a long time (I remember being slapped down on that very issue once). Let's try this Mac: $ mkdir -p fooo/baaa $ cd fooo/baaa $ chmod 0 .. $ /bin/pwd pwd: .: Permission denied $ So, yeah, not always reliable. This means that anything that presupplies names for [PHP-like __FILE__ and __DIR__] needs to compute it at need, not always. And the downside is that if these idoms, and importantly reliance on these idioms, becomes common then suddenly every Python program becomes a ticking time bomb for being run in an unusual-but-valid environment. Any kind of security conscious environment may require programs to run in constrained contexts. It is best to minimise the presumptions that a program makes in the name of convenience. More than once I've met devs whose code ran only in their (very very open/unsecured) dev environment and failed even in our staging environments. So I while it is rare, I don't think "/bin/pwd doesn't work" should be entirely ignored. Cheers, Cameron Simpson Every particle continues in its state of rest or uniform motion in a straight line except insofar as it doesn't. - Sir Arther Eddington From cs at zip.com.au Mon Apr 21 02:41:16 2014 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 21 Apr 2014 10:41:16 +1000 Subject: [Python-ideas] 'from os.path import FILE, DIR' or internal structure of filenames In-Reply-To: <20140420231244.GA36309@cskk.homeip.net> References: <20140420231244.GA36309@cskk.homeip.net> Message-ID: <20140421004116.GA69046@cskk.homeip.net> On this same topic... On 21Apr2014 09:12, Cameron Simpson wrote: >On 20Apr2014 14:41, anatoly techtonik wrote: >>>-1 because I can imagine wanting different nuances on the definitions >>> above; in particular for DIR I can well imagine wanting bare >>> dirname(abspath(FILE)) - semanticly different to your construction. >>> There's lots of scope for bikeshedding here. >> >>Assuming that FILE is always absolute, how that: >> dirname(abspath(FILE)) >>is different from that: >> abspath(dirname(FILE)) >>? > >Depends what "absolute" means. When the source file was obtained by >traversing a symlink, is FILE naive (ignored the symlink, just has to >start at "/") or >resolved (absolute path to FILE which does not traverse a symlink)? > >I can imagine use cases for both, and therefore the bikeshedding. For what it's worth, I am on the naive side of this argument. If we got to a file via a symlink or the like, then that is the filename "space" in which the app is expecting to run, and therefore we shouldn't muck with those expectations. Cheers, Cameron Simpson The supply chain is a network of atoms. - overhead by WIRED at the Intelligent Printing conference Oct2006 From 7vsfeu4pxg at snkmail.com Tue Apr 22 17:01:38 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Tue, 22 Apr 2014 17:01:38 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> Message-ID: <5047-1398178938-998242@sneakemail.com> On 20 April 2014 04:17, Bruce Leban bruce-at-leapyear.org | python-ideas-at-python.org| wrote: > > I found that the ugliest part of your proposal since it hides the first > case and at the same time makes it appear special when it isn't. > Myabe yes, but if you "translate" it in an if-elif, also the if seems to be "special". The indentation isn't required with my little Case class. The following > works just as well but lacks the visible scoping that the with statement > provides (of course the scope of the case variable leaks in either version, > but I could modify my class to fail if invoked after __exit__ or __del__).. > > for i in range(5): > print(i, end=' => ') > case = Case(i) > if case(1): > print('one') > elif case((2,3)): > > print('tuple(two, three)') > > elif case(2, 3): > print('two or three') > elif case > 3: > print('more than three') > > else: > > print('unmatched') > > Ok, but you have to nest it inside a for loop. Your class is a good solution, but a new syntax does not need nesting in a with or a for statement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Apr 22 17:15:51 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Apr 2014 01:15:51 +1000 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <5047-1398178938-998242@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> <5047-1398178938-998242@sneakemail.com> Message-ID: On Wed, Apr 23, 2014 at 1:01 AM, Lucas Malor <7vsfeu4pxg at snkmail.com> wrote: >> for i in range(5): >> print(i, end=' => ') >> case = Case(i) >> if case(1): >> print('one') >> elif case((2,3)): >> >> print('tuple(two, three)') >> >> elif case(2, 3): >> print('two or three') >> elif case > 3: >> print('more than three') >> >> else: >> >> print('unmatched') > > > Ok, but you have to nest it inside a for loop. Your class is a good > solution, but a new syntax does not need nesting in a with or a for > statement. > I don't think he *has* to nest it. My reading of the above is that it's the For-Case Paradigm [1], normally considered an anti-pattern but viable for demonstrating what happens in each case. ChrisA [1] http://thedailywtf.com/Articles/The_FOR-CASE_paradigm.aspx From 7vsfeu4pxg at snkmail.com Tue Apr 22 17:27:20 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Tue, 22 Apr 2014 17:27:20 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> <5047-1398178938-998242@sneakemail.com> Message-ID: <6710-1398180481-30395@sneakemail.com> Sorry, I didn't read it carefully. Anyway, youe example can be written as: for i in range(5): print(i, end=' => ') if i == 1: print('one') elif i == (2,3): print('tuple(two, three)') elif i in (2, 3): print('two or three') elif i > 3: print('more than three') else: print('unmatched') and it's much simpler to read. I suggested the switch statement for a simpler alternative to the if-elif chain. On 22 April 2014 17:15, Chris Angelico rosuav-at-gmail.com | python-ideas-at-python.org| wrote: > On Wed, Apr 23, 2014 at 1:01 AM, Lucas Malor <7vsfeu4pxg at snkmail.com> > wrote: > >> for i in range(5): > >> print(i, end=' => ') > >> case = Case(i) > >> if case(1): > >> print('one') > >> elif case((2,3)): > >> > >> print('tuple(two, three)') > >> > >> elif case(2, 3): > >> print('two or three') > >> elif case > 3: > >> print('more than three') > >> > >> else: > >> > >> print('unmatched') > > > > > > Ok, but you have to nest it inside a for loop. Your class is a good > > solution, but a new syntax does not need nesting in a with or a for > > statement. > > > > I don't think he *has* to nest it. My reading of the above is that > it's the For-Case Paradigm [1], normally considered an anti-pattern > but viable for demonstrating what happens in each case. > > ChrisA > > [1] http://thedailywtf.com/Articles/The_FOR-CASE_paradigm.aspx > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From 7vsfeu4pxg at snkmail.com Tue Apr 22 19:37:30 2014 From: 7vsfeu4pxg at snkmail.com (Lucas Malor) Date: Tue, 22 Apr 2014 19:37:30 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> <5047-1398178938-998242@sneakemail.com> <6710-1398180481-30395@sneakemail.com> Message-ID: <15206-1398188290-806777@sneakemail.com> On 22 April 2014 18:59, Bruce Leban bruce-at-leapyear.org |python-ideas-at-python.org| wrote: > My point is that this Case class does what you want with no language > changes. This is not correct. As I already said, I _would_ have a switch statement that is simpler to code than an if-elif chain. On the contrary your class creates code that is more complicated than an if-elif chain. Furthermore your class implements comparators other than "in" and "==", and this is not my goal. I quote what I wrote before: > I also thought about a syntax like this: > "case" comparator case_expr ":" > but IMHO it's too verbose for the typical uses of switch. If you want > flexibility, you can always use if-elif. And about simplicity, if I sacrifice the fallback and make the "break" the default and unique behaviour, I don't need "elcase" and my syntax will be more easy than before: switch tarot case 0: card = "Fool" case 1: card = "Alan Moore" case 2: card = "High Priestess" I think this is less complicated to read and it's more practical, since usually you want to "break". If you don't want to "break", you can create another switch. Probably I have to not use "switch" and "case", since it seems a C switch, while its behaviour is completely different now (C "case" falls back by default and can break, while this "case" breaks by default and can't fall back). Maybe something like dispatch-case, or inspect-case. From abarnert at yahoo.com Tue Apr 22 19:39:32 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Apr 2014 10:39:32 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <6710-1398180481-30395@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> <5047-1398178938-998242@sneakemail.com> <6710-1398180481-30395@sneakemail.com> Message-ID: <5E406DC6-9D49-44DC-8BF9-F9BB07762F2E@yahoo.com> On Apr 22, 2014, at 8:01, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > On 20 April 2014 04:17, Bruce Leban bruce-at-leapyear.org |python-ideas-at-python.org| wrote: >> >> I found that the ugliest part of your proposal since it hides the first case and at the same time makes it appear special when it isn't. > > Myabe yes, but if you "translate" it in an if-elif, also the if seems to be "special". Yes, there is a difference, but it's not nearly as dramatic as with your version. The conditions still look roughly parallel, only one short word away from the indent point, instead of the first one being way off to the right. On Apr 22, 2014, at 8:27, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > Sorry, I didn't read it carefully. Anyway, youe example can be written as: > > for i in range(5): > print(i, end=' => ') > if i == 1: > print('one') > elif i == (2,3): > print('tuple(two, three)') > elif i in (2, 3): > print('two or three') > elif i > 3: > print('more than three') > else: > print('unmatched') > > and it's much simpler to read. That's pretty much his point. The same can be said for all of your examples. Your version is only "simpler" in that your "case" and "elcase" are spelled "if case" and "elif case" in his. Meanwhile, your version is more complex in that the first case is crammed on the same line with the "switch". Also, instead of automatically handling tuples in a natural and obvious way, you're forced to invent a new builtin class and a new display form just to distinguish tuples inline in a case label, and still can't easily handle the distinction in a natural and readable way. And, while his can be trivially extended to handle other types of comparison besides == and in, yours can't. So, you've come up with something that's more complex, less natural, and less powerful than what we can already do, and the only benefit is a little bit of brevity. > I suggested the switch statement for a simpler alternative to the if-elif chain. > > > > On 22 April 2014 17:15, Chris Angelico rosuav-at-gmail.com |python-ideas-at-python.org| wrote: >> On Wed, Apr 23, 2014 at 1:01 AM, Lucas Malor <7vsfeu4pxg at snkmail.com> wrote: >> >> for i in range(5): >> >> print(i, end=' => ') >> >> case = Case(i) >> >> if case(1): >> >> print('one') >> >> elif case((2,3)): >> >> >> >> print('tuple(two, three)') >> >> >> >> elif case(2, 3): >> >> print('two or three') >> >> elif case > 3: >> >> print('more than three') >> >> >> >> else: >> >> >> >> print('unmatched') >> > >> > >> > Ok, but you have to nest it inside a for loop. Your class is a good >> > solution, but a new syntax does not need nesting in a with or a for >> > statement. >> > >> >> I don't think he *has* to nest it. My reading of the above is that >> it's the For-Case Paradigm [1], normally considered an anti-pattern >> but viable for demonstrating what happens in each case. >> >> ChrisA >> >> [1] http://thedailywtf.com/Articles/The_FOR-CASE_paradigm.aspx >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Apr 22 19:56:42 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Apr 2014 10:56:42 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20935-1397930131-420045@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20935-1397930131-420045@sneakemail.com> Message-ID: On Apr 19, 2014, at 10:54, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > On 19 April 2014 06:54, Andrew Barnert abarnert-at-yahoo.com |python-ideas-at-python.org| <3en9kh2cbt at sneakemail.com> wrote: >> switch mystring case "spam": >> process(thingy) >> >> That is not even remotely interpretable as an English sentence. That's not a problem for C, but it is for Python. > > I'll think about a possible solution. > > >>>> I don't see how skipping over any elcase but falling through to the next case is in any way simpler than C. >>> Well, because it's coherent with if-elif. See my last example in my first message. >> No it isn't. An elif clause is embedded in an if statement; there is no if clause you can embed on an if statement, just an entirely new and unrelated if statement. > > I can assert, without fear of contradictions, that my proposed syntax is the closest to the if-elif syntax of all the switch statement proposed until now in the age of Python. So what? All of those other syntaxes were rejected, and not because they weren't close enough to if-elif syntax. And besides, that's not actually true. Look at how if statements are defined in the grammar. You've attempted to match the syntax of a sequence of separate if statements within a single statement. > About off-topic arguments: > > >> bash does not have either ";;&" or ";&" > > http://www.gnu.org/software/bash/manual/html_node/Conditional-Constructs.html OK, I didn't know about this GNU extension. But it still doesn't answer any of my points. They are not the same as C break and continue, not are they the same as your proposed semantics. I won't repeat all the stuff you snipped out and didn't reply to, just the big one: you've cited bash and C as justification for your feature of skipping over some cases but then continuing to check others, but neither C not bash has any way to do that at all, much less a way that's similar to yours. >> Who asked you about list comprehensions or lambdas? What do you think they have to do with anything? >> [...] >> I have no idea why you think recursive functions are relevant to anything being discussed here. >> [...]your proposal was to treat tuples specially and match all other iterables as single values, which means that a generator expression would be matched as a single value, meaning it would only match itself. And again, I don't understand what the relevance is supposed to be. > > I premise that pattern recognition is outside the scope of my switch proposal. I already explained why it isn't. If you're looking to not just bring a case statement to Python, but invent a unique and more powerful one than any that have come before, you need to actually address what's come before. Especially since your syntax, by treating tuples specially, would prevent any future extensions along these lines, you need to explain why the multiple cases feature is more important than the pattern matching feature. > Anyway: > About list comprehensions and lambdas, you talked about assignment statements, and AFAIK the only pattern matching things that are somewhat related to assignment statement are the possibility to filter a list comprehension and the use of lambdas. If you mean something different you should be more explicit, instead of saying "what? where? why?" I still don't understand how you think filtering a list comprehension or using lambdas have anything to do with pattern matching, or what they have to do with assignment statements. Since you insist that you read up on what pattern matching means, I have no idea how to answer that. Especiallly since I already showed you exactly what I mean: x, y = expression() This requires the expression to return an iterable of exactly two values, and assigns those two values to x and y, in a way that's a analogous to (a duck-typed version of) ML-style pattern matching syntax. > About recursive function, I wronged. Sorry but it was late. > About generators, you can create a generator, convert it to an iterable and unpack it in a case_expr. I already explained that a generator is already an iterable and doesn't need to be converted into one. Your proposed syntax doesn't have any way to unpack iterables, and I'm not sure what it would do if you added such a thing (especially since you want to treat inline tuple displays specially, but not normal tuples, much less other iterables or sequences). > I could also extend the current syntax and support a "generator unpacking". This way you'll have a limited alternative to pattern matching in some cases in a switch statement without the need to create a static iterable. How does this provide an alternative to pattern matching? Show an example of what you mean. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leapyear.org Tue Apr 22 18:59:10 2014 From: bruce at leapyear.org (Bruce Leban) Date: Tue, 22 Apr 2014 09:59:10 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <6710-1398180481-30395@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> <5047-1398178938-998242@sneakemail.com> <6710-1398180481-30395@sneakemail.com> Message-ID: On Apr 22, 2014 8:28 AM, "Lucas Malor" <7vsfeu4pxg at snkmail.com> wrote: > > Sorry, I didn't read it carefully. Anyway, youe example can be written as: > >> for i in range(5): >> print(i, end=' => ') >> if i == 1: >> print('one') >> elif i == (2,3): >> print('tuple(two, three)') >> elif i in (2, 3): >> print('two or three') >> elif i > 3: >> print('more than three') >> else: >> print('unmatched') > > and it's much simpler to read. I suggested the switch statement for a simpler alternative to the if-elif chain. My code can be written as a series of static print statements too and that would be even easier to read. It's an *example* to illustrate use not actual useful code! My point is that this Case class does what you want with no language changes. And looking at your sample above exactly how would your new switch statement be better than either my code or your if/elif chain? --- Bruce (from my phone) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Tue Apr 22 21:35:23 2014 From: mertz at gnosis.cx (David Mertz) Date: Tue, 22 Apr 2014 12:35:23 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <15206-1398188290-806777@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> <5047-1398178938-998242@sneakemail.com> <6710-1398180481-30395@sneakemail.com> <15206-1398188290-806777@sneakemail.com> Message-ID: On Tue, Apr 22, 2014 at 10:37 AM, Lucas Malor <7vsfeu4pxg at snkmail.com>wrote: > switch tarot case 0: > card = "Fool" > case 1: > card = "Alan Moore" > case 2: > card = "High Priestess" > > -100. I have to say "yuck" to this sort of thing; it feels entirely unpythonic. Each 'case' is some sort of comparison, but you can't tell locally what is being compared. Of course in the simple one-line blocks it's easy enough to glance up to the 'switch', but if you had 20-30 lines of code within each 'case' you could look at 'case 20' and not remember/know what is being switched on. Using the Case class idea seems much more clear, as well as being much more flexible. It's true the initialization of the instance might be a ways up, but at least you know that it's an argument passed to a callable, so you already *know* that you'd have to look at the creation of that callable. Moreover, being generalizable is nice, e.g. (with a slightly enhanced class, implementation left to the reader): case = Case(my_color(some_complex_expression(this, that, other) + more_stuff(foo, bar)) if case('red'): some_red_stuff() elif case.in('green', 'blue'): some_bluegreen_stuff() elif '#00AAAA' < case < '#EEBBAA': # some definition of color range something_with_color_range() else: paint_it_black() Of course, in my example, maybe a better name for the 'case' instance is actually 'color'. The fact I can choose a name that fits the context adds even more clarity over all "cases" needing to be named 'case'. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Apr 22 21:44:34 2014 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 22 Apr 2014 20:44:34 +0100 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <5E406DC6-9D49-44DC-8BF9-F9BB07762F2E@yahoo.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140419031221.GI28400@ando> <20199-1397929416-249142@sneakemail.com> <5047-1398178938-998242@sneakemail.com> <6710-1398180481-30395@sneakemail.com> <5E406DC6-9D49-44DC-8BF9-F9BB07762F2E@yahoo.com> Message-ID: <5356C6A2.2050701@mrabarnett.plus.com> On 2014-04-22 18:39, Andrew Barnert wrote: > On Apr 22, 2014, at 8:01, "Lucas Malor" <7vsfeu4pxg at snkmail.com > > wrote: > >> On 20 April 2014 04:17, Bruce Leban bruce-at-leapyear.org >> |python-ideas-at-python.org >> | > > wrote: >> >> >> I found that the ugliest part of your proposal since it hides the >> first case and at the same time makes it appear special when it isn't. >> >> >> Myabe yes, but if you "translate" it in an if-elif, also the if seems >> to be "special". > > Yes, there is a difference, but it's not nearly as dramatic as with your > version. The conditions still look roughly parallel, only one short word > away from the indent point, instead of the first one being way off to > the right. > > On Apr 22, 2014, at 8:27, "Lucas Malor" <7vsfeu4pxg at snkmail.com > > wrote: > >> Sorry, I didn't read it carefully. Anyway, youe example can be written as: >> >> for i in range(5): >> print(i, end=' => ') >> if i == 1: >> print('one') >> elif i == (2,3): >> >> print('tuple(two, three)') >> >> elif i in (2, 3): >> print('two or three') >> elif i > 3: >> print('more than three') >> >> else: >> >> print('unmatched') >> >> >> and it's much simpler to read. > > That's pretty much his point. The same can be said for all of your examples. > > Your version is only "simpler" in that your "case" and "elcase" are > spelled "if case" and "elif case" in his. Meanwhile, your version is > more complex in that the first case is crammed on the same line with the > "switch". Also, instead of automatically handling tuples in a natural > and obvious way, you're forced to invent a new builtin class and a new > display form just to distinguish tuples inline in a case label, and > still can't easily handle the distinction in a natural and readable way. > And, while his can be trivially extended to handle other types of > comparison besides == and in, yours can't. So, you've come up with > something that's more complex, less natural, and less powerful than what > we can already do, and the only benefit is a little bit of brevity. > [snip] How about a slightly modified 'if' statement: for i in range(5): print(i, end=' => ') case if i == 1: print('one') elif (2, 3): # Same as elif == (2, 3) print('tuple(two, three)') elif in (2, 3): print('two or three') elif > 3: print('more than three') elif is None: print('None') elif == 0: # Or just elif 0 print('zero') else: print('unmatched') The 'case' says that you want to test the value of part preceding the comparator, in this case 'i'. In the 'elif' conditions, '==' is assumed unless you say otherwise. From jeanpierreda at gmail.com Wed Apr 23 00:06:55 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 22 Apr 2014 15:06:55 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> Message-ID: On Fri, Apr 18, 2014 at 9:54 PM, Andrew Barnert wrote: > Meanwhile, ML and friends let you go farther, matching by partial values: > > of (True, y): > process(y) > of Eggs(z): > fry(z) > > The first matches only if it's a tuple of exactly two values and the first > is equal to True, binding y to the second; the second matches only if it's > equal to an Eggs instance constructed with exactly one argument, and binds z > to that argument. > > Clearly not all of this fits into Python. (The last example isn't even > conceptually possible, given the way object initialization works.) But it's > worth considering what does and what doesn't fit rather than trying to > design a feature without even knowing what the options are. I'm really glad you brought this up, and it isn't conceptually impossible. Eggs(x) inside a pattern probably won't actually initialize an Eggs object; it probably would create a pattern object instead, and the value being matched can decide whether or not it obeys that pattern, and (if so) what object should be assigned to z. Why is it that every time someone brings up switch/case, they're being inspired by C and Java instead of ML or Haskell? People are looking backwards instead of forwards. Python doesn't have a nice way of pattern matching on tagged unions in general, not just C-style enums -- and it really ought to. I think pattern matching could fit in Python, where switch doesn't really. Switch replaces if/elif, pattern matching replaces the if/elif plus any work unpacking contents. The format of a pattern match encourages code to be written in a way that disjointly separates groups of disparate values, in a way that isn't difficult and doesn't involve weird type checks or whatever. Looking at the difference in e.g. AST manipulation code with/without is a nice example of how code can be made more readable. And looking at the problems Python code has today with this kind of thing is also illustrative -- for example, how do you tell the difference between no result, and a result whose value is None? In some APIs, you can't (!), in some, you need to specify a special sentinel object and check identity (ew), and in some, no result is represented by an exception -- which is fine, but in some cases very error prone (an uncaught StopIteration, for example, will terminate any generator that calls you, which can be difficult to notice.) Pattern matching offers a real alternative option here that wasn't realistic before. But, maybe this is one of those things that won't happen because Haskell licked the cookie and now everyone thinks it's gross and unreadable, and can't be convinced otherwise. I'm more worried that people don't even know this exists... -- Devin From phd at phdru.name Wed Apr 23 00:20:50 2014 From: phd at phdru.name (Oleg Broytman) Date: Wed, 23 Apr 2014 00:20:50 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> Message-ID: <20140422222050.GA3314@phdru.name> On Tue, Apr 22, 2014 at 03:06:55PM -0700, Devin Jeanpierre wrote: > Why is it that every time someone brings up switch/case, they're being > inspired by C and Java instead of ML or Haskell? How many people know imperative, procedural and OO languages? And how many know functional or logic-based languages? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From abarnert at yahoo.com Wed Apr 23 01:20:15 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Apr 2014 16:20:15 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> Message-ID: <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> On Apr 22, 2014, at 15:06, Devin Jeanpierre wrote: > On Fri, Apr 18, 2014 at 9:54 PM, Andrew Barnert > wrote: >> Meanwhile, ML and friends let you go farther, matching by partial values: >> >> of (True, y): >> process(y) >> of Eggs(z): >> fry(z) >> >> The first matches only if it's a tuple of exactly two values and the first >> is equal to True, binding y to the second; the second matches only if it's >> equal to an Eggs instance constructed with exactly one argument, and binds z >> to that argument. >> >> Clearly not all of this fits into Python. (The last example isn't even >> conceptually possible, given the way object initialization works.) But it's >> worth considering what does and what doesn't fit rather than trying to >> design a feature without even knowing what the options are. > > I'm really glad you brought this up, and it isn't conceptually > impossible. Eggs(x) inside a pattern probably won't actually > initialize an Eggs object; it probably would create a pattern object > instead, and the value being matched can decide whether or not it > obeys that pattern, and (if so) what object should be assigned to z. That's a promising idea that I'd love to see fleshed out. Forgive me for dumping a whole lot of questions on you as I think this through... First, does this really need to be restricted to case statements, or can constructor patterns just become another kind of target, which can be used anywhere (most importantly, assignment statements), and then cases just use the same targets? (Notice the potential parallels--and also differences--with __setattr__ and __setitem__ type assignments, not just tuple-style assignments.) I can see how this could work for complete matching. Let's say I assign spam = Spam(2, 3), and I later try to match that against Spam(x, y) where x and y are not bound names. So, Python calls spam.__match__(Spam, 'x', 'y'), which returns {'x': 2, 'y': 3}, so the case matches and x and y get locally bound to 2 and 3. But what about partial matching? What if I want to match against Spam(2, y)? If you want that to work with literals, it might be possible. (Note that "2, y = eggs" is caught by the early stages of the compiler today, because the grammar doesn't allow literals in assignment targets.) Maybe something like spam.__match__(Spam, (2, True), ('y', False))? That's not conceptually too hard, but it would get really ugly inside the __match__ method of every class, unless you had some good types and helper functions similar to what you have in the opposite direction with inspect.Signature.bind. If you want partial matching with dynamic values, however, it's a lot harder. You can't put expressions in target lists today, for obvious reasons (they have a partially parallel but simpler grammar), and targets are never treated as values directly. In particular, "x = 2" doesn't raise a NoItIsntError if x is already bound to 1, it just rebinds it. This means that if we do have partial patterns, they may still not be the same as (or as powerful as) you'd like. For example, if matching Spam(x, y) always matches and binds x as a name, rather than partially matching on its value (unless you can think of some syntax for that?) it's a lot harder to write dynamic patterns. (You can probably do something with partial, but that just makes matching even more complicated and leaves the syntax for using it in these cases unfriendly.) Also, keep in mind that tuple-like targets always expand an iterable into a list, so "x, *y = range(3)" matches 0 and [1, 2], and "x, *y = itertools.count()" raises a MemoryError, so you're never going to have the full power of Haskell patterns here. Beyond that, is there anything else to add to target lists besides call-like forms to match constructor patterns, or is that sufficient for everything you need from pattern matching? > Why is it that every time someone brings up switch/case, they're being > inspired by C and Java instead of ML or Haskell? People are looking > backwards instead of forwards. I agree, but it's certainly more complex, and harder to fit into Python. Thanks for showing me that it's not actually intractable, though... > Python doesn't have a nice way of > pattern matching on tagged unions in general, not just C-style enums > -- and it really ought to. I think pattern matching could fit in > Python, where switch doesn't really. Switch replaces if/elif, pattern > matching replaces the if/elif plus any work unpacking contents. The > format of a pattern match encourages code to be written in a way that > disjointly separates groups of disparate values, in a way that isn't > difficult and doesn't involve weird type checks or whatever. > > Looking at the difference in e.g. AST manipulation code with/without > is a nice example of how code can be made more readable. And looking > at the problems Python code has today with this kind of thing is also > illustrative -- for example, how do you tell the difference between no > result, and a result whose value is None? In some APIs, you can't (!), > in some, you need to specify a special sentinel object and check > identity (ew), and in some, no result is represented by an exception > -- which is fine, but in some cases very error prone (an uncaught > StopIteration, for example, will terminate any generator that calls > you, which can be difficult to notice.) Pattern matching offers a > real alternative option here that wasn't realistic before. I think coming up with just enough of a proposal to then show how you could rewrite some existing AST (or DOM?) manipulation code, might make the case a lot better than trying to explain it. If you're not up for doing that, I might try to tackle it, but it's your idea and you're the one with use cases in his head. > But, maybe this is one of those things that won't happen because > Haskell licked the cookie and now everyone thinks it's gross and > unreadable, and can't be convinced otherwise. I'm more worried that > people don't even know this exists... The main objection to bringing in declarative features from functional languages, especially Haskell, is that it often forces people to read things at a higher level of abstraction. That isn't true for all such features (list comprehensions, anyone?), but it's a default assumption you have to overcome. I think you can overcome it here; reading the actual __match__ methods is one thing, but the (presumably much more common, especially in novice code) uses of case statements or other matching targets can be just as explicit and concrete as if-based switching code, and a lot more readable. We just need to make that case for the specific feature strongly enough to overcome the general presumption. I think your AST example would go a long way toward doing that. From abarnert at yahoo.com Wed Apr 23 01:29:02 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Apr 2014 16:29:02 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20140422222050.GA3314@phdru.name> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140422222050.GA3314@phdru.name> Message-ID: <73FE9F65-AAB1-4332-B2F0-AB6967D477CD@yahoo.com> On Apr 22, 2014, at 15:20, Oleg Broytman wrote: > On Tue, Apr 22, 2014 at 03:06:55PM -0700, Devin Jeanpierre wrote: >> Why is it that every time someone brings up switch/case, they're being >> inspired by C and Java instead of ML or Haskell? > > How many people know imperative, procedural and OO languages? And how > many know functional or logic-based languages? I think most people know multi-paradigm languages. C++ has similar features, even if many of them are restricted to it's compile-time template engine. Most JavaScript programs are all about passing anonymous functions to higher-order functions as callbacks, of abstractions on top of that like futures. And so on. Anyway, people don't have a problem learning map, any, comprehensions, unpacking, or other functional-derived features in Python even if they come from C; in fact, those features are often why they don't want to go back to C. The issue is always finding the right balance, borrowing features that fit into the language well and make code easier to read but not trying to encourage recursive folds over loops or rebuild I/O around monads or pure but nondeterministic functions. From haoyi.sg at gmail.com Wed Apr 23 01:38:16 2014 From: haoyi.sg at gmail.com (Haoyi Li) Date: Tue, 22 Apr 2014 16:38:16 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <73FE9F65-AAB1-4332-B2F0-AB6967D477CD@yahoo.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <20140422222050.GA3314@phdru.name> <73FE9F65-AAB1-4332-B2F0-AB6967D477CD@yahoo.com> Message-ID: Fun fact: we've had pattern matching on 2.7 using MacroPy for a year now https://github.com/lihaoyi/macropy#pattern-matching @caseclass Nil(): pass @caseclass Cons(x, xs): pass def reduce(op, my_list): with switch(my_list): if Cons(x, Nil()): return x elif Cons(x, xs): return op(x, reduce(op, xs)) On Tue, Apr 22, 2014 at 4:29 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Apr 22, 2014, at 15:20, Oleg Broytman wrote: > > > On Tue, Apr 22, 2014 at 03:06:55PM -0700, Devin Jeanpierre < > jeanpierreda at gmail.com> wrote: > >> Why is it that every time someone brings up switch/case, they're being > >> inspired by C and Java instead of ML or Haskell? > > > > How many people know imperative, procedural and OO languages? And how > > many know functional or logic-based languages? > > I think most people know multi-paradigm languages. C++ has similar > features, even if many of them are restricted to it's compile-time template > engine. Most JavaScript programs are all about passing anonymous functions > to higher-order functions as callbacks, of abstractions on top of that like > futures. And so on. > > Anyway, people don't have a problem learning map, any, comprehensions, > unpacking, or other functional-derived features in Python even if they come > from C; in fact, those features are often why they don't want to go back to > C. The issue is always finding the right balance, borrowing features that > fit into the language well and make code easier to read but not trying to > encourage recursive folds over loops or rebuild I/O around monads or pure > but nondeterministic functions. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Wed Apr 23 05:58:46 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 22 Apr 2014 20:58:46 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> Message-ID: On Tue, Apr 22, 2014 at 4:20 PM, Andrew Barnert wrote: > On Apr 22, 2014, at 15:06, Devin Jeanpierre wrote: >> I'm really glad you brought this up, and it isn't conceptually >> impossible. Eggs(x) inside a pattern probably won't actually >> initialize an Eggs object; it probably would create a pattern object >> instead, and the value being matched can decide whether or not it >> obeys that pattern, and (if so) what object should be assigned to z. > > That's a promising idea that I'd love to see fleshed out. Forgive me for dumping a whole lot of questions on you as I think this through... > > First, does this really need to be restricted to case statements, or can constructor patterns just become another kind of target, which can be used anywhere (most importantly, assignment statements), and then cases just use the same targets? (Notice the potential parallels--and also differences--with __setattr__ and __setitem__ type assignments, not just tuple-style assignments.) Making assignment statements do a lot of work would slow down Python way too much. The restricted sort of destructuring bind python already does is sufficiently limited that the compiler can do all the heavy lifting. > I can see how this could work for complete matching. Let's say I assign spam = Spam(2, 3), and I later try to match that against Spam(x, y) where x and y are not bound names. So, Python calls spam.__match__(Spam, 'x', 'y'), which returns {'x': 2, 'y': 3}, so the case matches and x and y get locally bound to 2 and 3. > > But what about partial matching? What if I want to match against Spam(2, y)? If you want that to work with literals, it might be possible. (Note that "2, y = eggs" is caught by the early stages of the compiler today, because the grammar doesn't allow literals in assignment targets.) Maybe something like spam.__match__(Spam, (2, True), ('y', False))? That's not conceptually too hard, but it would get really ugly inside the __match__ method of every class, unless you had some good types and helper functions similar to what you have in the opposite direction with inspect.Signature.bind. That's a bit complicated, and another example of when a simpleish API is made hard without nice union types. Why not use pattern matching? :) spam.__match__(ConstructorPattern(Spam, Var('x'), Var('y'))) and spam.__match__(ConstructorPattern(Spam, Literal(2), Var('y'))) in __match__ you can check "match pat: case ConstructorPattern(t, Var(a), Var(b)): if t == Spam: ...", etc. (This is not cyclic; the base cases are the __match__ for ConstructorPattern, Var, and Literal, which can do some difficult manual work instead). In most cases people would not be writing __match__ by hand, so it doesn't need to be easy to write. You'd probably be defining matchable cases using something like what Haoyi Li wrote, except with a namedtuple-analogue instead of AST macros: Nil = constructor('Nil', ()) Cons = constructor('Cons', ('x', 'xs')) value = Cons('hello world', Nil()) match value: case Cons(a, Nil()): print(a) # a == value.x == 'hello world' > If you want partial matching with dynamic values, however, it's a lot harder. You can't put expressions in target lists today, for obvious reasons (they have a partially parallel but simpler grammar), and targets are never treated as values directly. In particular, "x = 2" doesn't raise a NoItIsntError if x is already bound to 1, it just rebinds it. This means that if we do have partial patterns, they may still not be the same as (or as powerful as) you'd like. For example, if matching Spam(x, y) always matches and binds x as a name, rather than partially matching on its value (unless you can think of some syntax for that?) it's a lot harder to write dynamic patterns. (You can probably do something with partial, but that just makes matching even more complicated and leaves the syntax for using it in these cases unfriendly.) I am not sure what you're getting at here, exactly. x should never be asked if it's OK to do x = 2. If you want to use variables as values to compare with, rather than targets to assign to, usually that's done by adding a conditional. Although, Racket has a match extension for using a variable to check for equality in a pattern match, like so: > (define (check_eq x y) (match x [ (== y) (displayln "x == y")] [ _ (displayln "x != y")])) > (check_eq 1 1) x == y > (check_eq 1 2) x != y One could add such a thing (maybe $x instead of (== x) ? ;O), or use guard clauses, or just use a nested if. > Also, keep in mind that tuple-like targets always expand an iterable into a list, so "x, *y = range(3)" matches 0 and [1, 2], and "x, *y = itertools.count()" raises a MemoryError, so you're never going to have the full power of Haskell patterns here. > > Beyond that, is there anything else to add to target lists besides call-like forms to match constructor patterns, or is that sufficient for everything you need from pattern matching? You haven't presented an organized list :( I think a good set of features is: - variables (e.g. matching against x) - constructor/term patterns (e.g. matching against foo(a, b)) - literals (e.g. matching against (a, b) or even (a, 2)) - repetition operator (e.g. matching against *a) # nonessential, but powerful - combining patterns (e.g. match against foo(a) or bar(a, 2)) # convenient - binding subpatterns to names (e.g. match against foo(bar(z) as x); now x equals the thing that matched bar(z)) # convenient And take the transitive closure of combining those operations. That's a pretty big wishlist, with implications that maybe would weird people out, so I won't explain the details. They aren't important. > I think coming up with just enough of a proposal to then show how you could rewrite some existing AST (or DOM?) manipulation code, might make the case a lot better than trying to explain it. I doubt such a proposal would be accepted, but I am OK writing what I did above and showing an example. compare http://hg.python.org/cpython/file/v2.7.3/Lib/ast.py#l52 with: def _convert(node): match node: case Str(v) or Num(v): return v case Tuple(vs): return tuple(map(_convert, vs)) case List(vs): return list(map(_convert, vs)) case Dict(_): return dict((_convert(k), _convert(v)) for k, v in zip(node.keys, node.values)) case Name(id): if id in _safe_names: return _safe_names[id] case BinOp(Num(left), op, Num(right)): if isinstance(left, (int,long,float)) and isinstance(right, (int,long,float))): match op: case Add(): return left + right case Sub(): return left - right case _: pass # presumably failure to match is an error? idk raise ValueError('malformed string') This is definitely more readable, in particular in the BinOp case. And you'll note that a mere switch statement really isn't very helpful. ;) -- Devin From jeanpierreda at gmail.com Wed Apr 23 08:13:44 2014 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 22 Apr 2014 23:13:44 -0700 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> Message-ID: On Tue, Apr 22, 2014 at 8:58 PM, Devin Jeanpierre wrote: > On Tue, Apr 22, 2014 at 4:20 PM, Andrew Barnert wrote: >> On Apr 22, 2014, at 15:06, Devin Jeanpierre wrote: >>> I'm really glad you brought this up, and it isn't conceptually >>> impossible. Eggs(x) inside a pattern probably won't actually >>> initialize an Eggs object; it probably would create a pattern object >>> instead, and the value being matched can decide whether or not it >>> obeys that pattern, and (if so) what object should be assigned to z. >> >> That's a promising idea that I'd love to see fleshed out. Forgive me for dumping a whole lot of questions on you as I think this through... >> >> First, does this really need to be restricted to case statements, or can constructor patterns just become another kind of target, which can be used anywhere (most importantly, assignment statements), and then cases just use the same targets? (Notice the potential parallels--and also differences--with __setattr__ and __setitem__ type assignments, not just tuple-style assignments.) > > Making assignment statements do a lot of work would slow down Python > way too much. The restricted sort of destructuring bind python already > does is sufficiently limited that the compiler can do all the heavy > lifting. > >> I can see how this could work for complete matching. Let's say I assign spam = Spam(2, 3), and I later try to match that against Spam(x, y) where x and y are not bound names. So, Python calls spam.__match__(Spam, 'x', 'y'), which returns {'x': 2, 'y': 3}, so the case matches and x and y get locally bound to 2 and 3. >> >> But what about partial matching? What if I want to match against Spam(2, y)? If you want that to work with literals, it might be possible. (Note that "2, y = eggs" is caught by the early stages of the compiler today, because the grammar doesn't allow literals in assignment targets.) Maybe something like spam.__match__(Spam, (2, True), ('y', False))? That's not conceptually too hard, but it would get really ugly inside the __match__ method of every class, unless you had some good types and helper functions similar to what you have in the opposite direction with inspect.Signature.bind. > > That's a bit complicated, and another example of when a simpleish API > is made hard without nice union types. Why not use pattern matching? > :) > > spam.__match__(ConstructorPattern(Spam, Var('x'), Var('y'))) and > spam.__match__(ConstructorPattern(Spam, Literal(2), Var('y'))) > > in __match__ you can check "match pat: case ConstructorPattern(t, > Var(a), Var(b)): if t == Spam: ...", etc. (This is not cyclic; the > base cases are the __match__ for ConstructorPattern, Var, and Literal, > which can do some difficult manual work instead). Sorry, this is unsound / doesn't specify where the recursion is allowed to happen. Instead, have __match__ accept a constructor and a list of identifiers and return a dict mapping identifiers to values -- it never sees anything other than variables in the constructor parameters. (No literals.) If you try to match foo with Spam(2, y), then __match__ gets Spam, generated_variable_12345, and y. When it fills in generated_variable_12345, the thing it assigns to that variable is subsequently pattern matched against 2 (probably using __eq__ instead of an overloaded __match__). Stupid mistake I make a lot when I'm not thinking about recursive algorithms... :( -- Devin > In most cases people would not be writing __match__ by hand, so it > doesn't need to be easy to write. You'd probably be defining matchable > cases using something like what Haoyi Li wrote, except with a > namedtuple-analogue instead of AST macros: > > Nil = constructor('Nil', ()) > Cons = constructor('Cons', ('x', 'xs')) > value = Cons('hello world', Nil()) > match value: > case Cons(a, Nil()): > print(a) # a == value.x == 'hello world' > > >> If you want partial matching with dynamic values, however, it's a lot harder. You can't put expressions in target lists today, for obvious reasons (they have a partially parallel but simpler grammar), and targets are never treated as values directly. In particular, "x = 2" doesn't raise a NoItIsntError if x is already bound to 1, it just rebinds it. This means that if we do have partial patterns, they may still not be the same as (or as powerful as) you'd like. For example, if matching Spam(x, y) always matches and binds x as a name, rather than partially matching on its value (unless you can think of some syntax for that?) it's a lot harder to write dynamic patterns. (You can probably do something with partial, but that just makes matching even more complicated and leaves the syntax for using it in these cases unfriendly.) > > I am not sure what you're getting at here, exactly. x should never be > asked if it's OK to do x = 2. If you want to use variables as values > to compare with, rather than targets to assign to, usually that's done > by adding a conditional. Although, Racket has a match extension for > using a variable to check for equality in a pattern match, like so: > >> (define (check_eq x y) > (match x > [ (== y) (displayln "x == y")] > [ _ (displayln "x != y")])) >> (check_eq 1 1) > x == y >> (check_eq 1 2) > x != y > > One could add such a thing (maybe $x instead of (== x) ? ;O), or use > guard clauses, or just use a nested if. > >> Also, keep in mind that tuple-like targets always expand an iterable into a list, so "x, *y = range(3)" matches 0 and [1, 2], and "x, *y = itertools.count()" raises a MemoryError, so you're never going to have the full power of Haskell patterns here. >> >> Beyond that, is there anything else to add to target lists besides call-like forms to match constructor patterns, or is that sufficient for everything you need from pattern matching? > > You haven't presented an organized list :( > > I think a good set of features is: > > - variables (e.g. matching against x) > - constructor/term patterns (e.g. matching against foo(a, b)) > - literals (e.g. matching against (a, b) or even (a, 2)) > - repetition operator (e.g. matching against *a) # nonessential, but powerful > - combining patterns (e.g. match against foo(a) or bar(a, 2)) # convenient > - binding subpatterns to names (e.g. match against foo(bar(z) as x); > now x equals the thing that matched bar(z)) # convenient > > And take the transitive closure of combining those operations. That's > a pretty big wishlist, with implications that maybe would weird people > out, so I won't explain the details. They aren't important. > >> I think coming up with just enough of a proposal to then show how you could rewrite some existing AST (or DOM?) manipulation code, might make the case a lot better than trying to explain it. > > I doubt such a proposal would be accepted, but I am OK writing what I > did above and showing an example. > > compare http://hg.python.org/cpython/file/v2.7.3/Lib/ast.py#l52 with: > > def _convert(node): > match node: > case Str(v) or Num(v): > return v > case Tuple(vs): > return tuple(map(_convert, vs)) > case List(vs): > return list(map(_convert, vs)) > case Dict(_): > return dict((_convert(k), _convert(v)) for k, v > in zip(node.keys, node.values)) > case Name(id): > if id in _safe_names: > return _safe_names[id] > case BinOp(Num(left), op, Num(right)): > if isinstance(left, (int,long,float)) and > isinstance(right, (int,long,float))): > match op: > case Add(): > return left + right > case Sub(): > return left - right > case _: pass # presumably failure to match is an error? idk > > raise ValueError('malformed string') > > This is definitely more readable, in particular in the BinOp case. And > you'll note that a mere switch statement really isn't very helpful. ;) > > -- Devin From pconnell at gmail.com Wed Apr 23 10:38:03 2014 From: pconnell at gmail.com (Phil Connell) Date: Wed, 23 Apr 2014 09:38:03 +0100 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> Message-ID: <20140423083803.GA7418@phconnel-ws.cisco.com> On Tue, Apr 22, 2014 at 11:13:44PM -0700, Devin Jeanpierre wrote: > On Tue, Apr 22, 2014 at 8:58 PM, Devin Jeanpierre > wrote: > > On Tue, Apr 22, 2014 at 4:20 PM, Andrew Barnert wrote: > >> On Apr 22, 2014, at 15:06, Devin Jeanpierre wrote: > >> I can see how this could work for complete matching. Let's say I assign spam = Spam(2, 3), and I later try to match that against Spam(x, y) where x and y are not bound names. So, Python calls spam.__match__(Spam, 'x', 'y'), which returns {'x': 2, 'y': 3}, so the case matches and x and y get locally bound to 2 and 3. > >> > >> But what about partial matching? What if I want to match against Spam(2, y)? If you want that to work with literals, it might be possible. (Note that "2, y = eggs" is caught by the early stages of the compiler today, because the grammar doesn't allow literals in assignment targets.) Maybe something like spam.__match__(Spam, (2, True), ('y', False))? That's not conceptually too hard, but it would get really ugly inside the __match__ method of every class, unless you had some good types and helper functions similar to what you have in the opposite direction with inspect.Signature.bind. > > > > That's a bit complicated, and another example of when a simpleish API > > is made hard without nice union types. Why not use pattern matching? > > :) > > > > spam.__match__(ConstructorPattern(Spam, Var('x'), Var('y'))) and > > spam.__match__(ConstructorPattern(Spam, Literal(2), Var('y'))) > > > > in __match__ you can check "match pat: case ConstructorPattern(t, > > Var(a), Var(b)): if t == Spam: ...", etc. (This is not cyclic; the > > base cases are the __match__ for ConstructorPattern, Var, and Literal, > > which can do some difficult manual work instead). > > Sorry, this is unsound / doesn't specify where the recursion is > allowed to happen. Instead, have __match__ accept a constructor and a > list of identifiers and return a dict mapping identifiers to values -- > it never sees anything other than variables in the constructor > parameters. (No literals.) > > If you try to match foo with Spam(2, y), then __match__ gets Spam, > generated_variable_12345, and y. When it fills in > generated_variable_12345, the thing it assigns to that variable is > subsequently pattern matched against 2 (probably using __eq__ instead > of an overloaded __match__). This looks promising :) I see two possibilities for the match protocol: 1. Have a single __match__ classmethod that is responsible for 'binding' unbound values, and work out whether there is a match *outside* of the __match__ method (using __eq__). 2. Split the protocol into two methods: __match_check__, responsible for checking whether there is a match. __match_unpack__, responsible for extracting the match parameters of an object. I've got a strong suspicion that #2 would turn out to be the better solution: - It should be more efficient: __match_check__ can short-circuit after the first non-matching parameter, whereas a standalone __match__ would have to unpack/bind all parameters. - It allows greater flexibility, for example fuzzy matching: is the value in an allowed range? - I *think* the methods would probably have simpler implementations (ultimate proof needs code :) The protocol methods would look something like this: class Spam: @classmethod def __match_check__(cls, other, a, b, c): ... @classmethod def __match_unpack__(cls, other): Used something like this: match foo: case Spam(1, 2, 3): ... case Spam(42, x, y): ... Semantics of __match_check__: - 'other' is the object being matched (whatever 'foo' refers to). - a, b, c come directly from 'case Spam(...)', i.e. 1, 2, 3 in the first case. - If some of the args are 'unbound' the corresponding parameters refer to a new Unbound singleton (better name?) , i.e. in the second case a=42, b=Unbound, c=Unbound. - Should return True if 'other' matches given the parameters' values, False otherwise. Semantics of __match_unpack__: - 'other' is the object being matched, again. - The return signature is less obvious. Discussion below. - From the returned value, unbound parameters are filled in (i.e. x and y in the second case). Options for the return value of __match_unpack__: A. Simplest case would be a tuple, something like (other.a, other.b, other.c) in the example. B. More flexible would be a pair (values, kwvalues) where 'values' is an iterable of 'positional' matchable values, and 'kwvalues' is a dict of 'keyword' matchable values. This implies a more complex case syntax, e.g. case Eggs(a as x) would extract the kwvalue 'a' and bind it to the name 'x' in the case block. Not sure this is worth it. C. Have __match_unpack__ accept *args, filled in with strings giving the names of matchable parameters to be unpacked? (Not convinced this is a good idea :) Finally, one thing that's worth noting: this only needs language support to allow 'unbound' values in case statements. It should be completely possible to implement this as a third party module with a couple of differences: - 'case Spam(x, y)' would need to be spelt 'elif case(Spam(Var("x"), Var("y")))' - The match/case syntactic sugar would obviously not work ('with Case() as case' etc needed instead). Cheers, Phil From g.rodola at gmail.com Wed Apr 23 23:35:52 2014 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 23 Apr 2014 23:35:52 +0200 Subject: [Python-ideas] Design decisions about socket.sendfile() Message-ID: In http://bugs.python.org/issue17552 me and a bunch of other devs are currently discussing what kind of API socket.sendfile() should have and because it seems there are different opinions about it I though I should have brought this up here for further discussion. With issue 17552 I'm proposing to provide a wrapper on top of os.sendfile(), socket.send() and possibly TransmitFile on Windows. AFAIK the modules which could immediately benefit from this addition are ftplib ( http://bugs.python.org/issue13564) and httplib ( http://bugs.python.org/issue13559). This is the current function signature as of socket-sendfile5.patch: def sendfile(self, file, blocksize=262144, offset=0, use_fallback=True): """sendfile(file[, blocksize[, offset[, use_fallback]]]) -> sent Send a file until EOF is reached attempting to use high-performance os.sendfile() in which case *file* must be a regular file object opened in binary mode; if not and *use_fallback* is True send() will be used instead. File position is updated on return or also in case of error in which case file.tell() can be used to figure out the number of bytes which were transmitted. *blocksize* is the maximum number of bytes to transmit at one time, *offset* tells from where to start reading the file. The socket must be of SOCK_STREAM type. Non-blocking sockets are not supported. Return the total number of bytes which were transmitted. """ Debatable questions which were raised during the discussion on the bug tracker are: 1 - whether to provide a "use_fallback" argument which when False raises an exception in case os.sendfile() could not be used 2 - whether to provide a custom exception in order to signal the number of bytes transmitted as opposed to use file.tell() afterwards; for the record, I'm -1 on this. 3 - whether to commit the patch as-is without including Windows support via TransmitFile() and post-pone that for a later time in a separate issue 4 - (extra) whether to provide a "callback" argument which gets called against each block of data before it is being sent. This would be useful to apply data transformation or to implement a progress bar or something. Note if this gets added it would sort of conflict with the "use_fallback=False" parameter because in order to apply data transformation socket.send() must be used instead of os.sendfile(). So far #1 appears to be the most debatable question. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Apr 25 07:11:05 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 25 Apr 2014 07:11:05 +0200 Subject: [Python-ideas] Python-specific Travis CI In-Reply-To: <8FF25312-584E-4C52-939B-526C2473E2F4@stufft.io> References: <0eab03d4b4d520ef48419ab47270e281@hmmz.org> <60C9CDE8-36FC-4534-A12A-4A81351DE0F7@stufft.io> <96949f3566ab45335852c172c7bdf970@hmmz.org> <8FF25312-584E-4C52-939B-526C2473E2F4@stufft.io> Message-ID: Donald Stufft, 20.04.2014 03:34: > On Apr 19, 2014, at 6:51 PM, David Wilson wrote: >> Travis can work well, but the effort involved in maintaining it has, at least for me, been unjustifiably high: in total I've spent close to a man-week in the past 6 months trying various configurations, and fixing things up once a month, before resorting to what I have now. In all, I have probably spent 10x the time maintaining Travis as I have actually writing comprehensive tests. >> It cannot reasonably be expected that project maintainers should pay similarly, or ideally even need to be aware their package is undergoing testing someplace. >> My biggest gripe with Travis, though, is that they can and do remove things that break what for me seems trivial functionality. This is simply the nature of a one-size-fits-all service. From what I gather, they previously removed old Python releases from their base image to reduce its size. > > I?ve not had anywhere near that type of experience trying to get anything setup on Travis. I'm not an overly happy Travis user myself, but I have to concur that the maintenance hasn't been a burden at all for me, in neither of three Python projects that I let them test. I'm also not sure a Python specific testing service would end up being successful. It's a lot of work to set up (snakebite?), and then to maintain and keep running, keep it safe and reliable, etc. Anyone who considers giving it a try should assume it to become a full time job for at least a couple of months, and only then take the decision. Stefan From flying-sheep at web.de Sat Apr 26 15:03:32 2014 From: flying-sheep at web.de (Philipp A.) Date: Sat, 26 Apr 2014 15:03:32 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20140423083803.GA7418@phconnel-ws.cisco.com> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: what y?all are missing about switch/case is that it?s *not* necessarily a series of comparisons in most languages. in essence, it?s a hashmap lookup with (in some languages) optional comparisons. in python, the most similar thing is something like this: def branch_2_3(): spam(1) spam(6) { 'key1': lambda: eggs += lay_eggs(), #branch 1 'key2': branch_2_3, 'key3': branch_2_3, }[value]() which is of course completely ugly and unnecessarily complicated for cases where multiple keys lead to the same branch. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 26 15:27:47 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 26 Apr 2014 23:27:47 +1000 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: On Sat, Apr 26, 2014 at 11:03 PM, Philipp A. wrote: > in python, the most similar thing is something like this: > > 'key1': lambda: eggs += lay_eggs(), #branch 1 Except that lambda doesn't create a code block, it creates a nested function. This example won't work, and nor will anything else that needs to mutate locals; you'd have to declare an out-of-line function (as with branch_2_3) to be able to assign to 'nonlocal eggs'. ChrisA From flying-sheep at web.de Sat Apr 26 15:43:27 2014 From: flying-sheep at web.de (Philipp A.) Date: Sat, 26 Apr 2014 15:43:27 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: sure it works if `eggs` has a `__iadd__` method. why shouldn?t it use the outer local? but that?s irrelevant. as i said: what i did is closest thing. yet in some other respects, if/else is closer, as there is no real alternative to a switch statement in python. my only point is that apart from being cleaner, a switch block also behaves differently than a if/else chain, and performs better. 2014-04-26 15:27 GMT+02:00 Chris Angelico : > On Sat, Apr 26, 2014 at 11:03 PM, Philipp A. wrote: > > in python, the most similar thing is something like this: > > > > 'key1': lambda: eggs += lay_eggs(), #branch 1 > > Except that lambda doesn't create a code block, it creates a nested > function. This example won't work, and nor will anything else that > needs to mutate locals; you'd have to declare an out-of-line function > (as with branch_2_3) to be able to assign to 'nonlocal eggs'. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 26 15:54:19 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 26 Apr 2014 23:54:19 +1000 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: On Sat, Apr 26, 2014 at 11:43 PM, Philipp A. wrote: > sure it works if `eggs` has a `__iadd__` method. why shouldn?t it use the > outer local? 1) Operator precedence gets in the way. (Easily fixed.) >>> lambda: eggs += lay_eggs() SyntaxError: can't assign to lambda 2) Assignment is a statement, so you can't do it in a lambda. >>> lambda: (eggs += lay_eggs()) SyntaxError: invalid syntax 3) Assignment makes it local, so you'll get UnboundLocalError if you don't declare it nonlocal. >>> def f(): eggs += lay_eggs() >>> f() Traceback (most recent call last): File "", line 1, in f() File "", line 2, in f eggs += lay_eggs() UnboundLocalError: local variable 'eggs' referenced before assignment Hence my previous statement that you need to write the function out of line, which breaks the switch-ness of the block, and you definitely need to declare eggs nonlocal. Now, if eggs is a list, you can extend it with a method (which can be called in an expression), but if your switch block can only do expressions, it's pretty limited. ChrisA From flying-sheep at web.de Sat Apr 26 16:44:51 2014 From: flying-sheep at web.de (Philipp A.) Date: Sat, 26 Apr 2014 16:44:51 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: interesting. it?s still assignment, even if nothing gets assigned (and only __iadd__ gets called behind the scenes). 2014-04-26 15:54 GMT+02:00 Chris Angelico : > On Sat, Apr 26, 2014 at 11:43 PM, Philipp A. wrote: > > sure it works if `eggs` has a `__iadd__` method. why shouldn?t it use the > > outer local? > > 1) Operator precedence gets in the way. (Easily fixed.) > >>> lambda: eggs += lay_eggs() > SyntaxError: can't assign to lambda > > 2) Assignment is a statement, so you can't do it in a lambda. > >>> lambda: (eggs += lay_eggs()) > SyntaxError: invalid syntax > > 3) Assignment makes it local, so you'll get UnboundLocalError if you > don't declare it nonlocal. > >>> def f(): > eggs += lay_eggs() > >>> f() > Traceback (most recent call last): > File "", line 1, in > f() > File "", line 2, in f > eggs += lay_eggs() > UnboundLocalError: local variable 'eggs' referenced before assignment > > Hence my previous statement that you need to write the function out of > line, which breaks the switch-ness of the block, and you definitely > need to declare eggs nonlocal. Now, if eggs is a list, you can extend > it with a method (which can be called in an expression), but if your > switch block can only do expressions, it's pretty limited. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sat Apr 26 16:58:27 2014 From: ron3200 at gmail.com (Ron Adam) Date: Sat, 26 Apr 2014 10:58:27 -0400 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: On 04/26/2014 09:54 AM, Chris Angelico wrote: > On Sat, Apr 26, 2014 at 11:43 PM, Philipp A. wrote: >> >sure it works if `eggs` has a `__iadd__` method. why shouldn?t it use the >> >outer local? > 1) Operator precedence gets in the way. (Easily fixed.) >>>> >>>lambda: eggs += lay_eggs() > SyntaxError: can't assign to lambda > > 2) Assignment is a statement, so you can't do it in a lambda. >>>> >>>lambda: (eggs += lay_eggs()) > SyntaxError: invalid syntax > > 3) Assignment makes it local, so you'll get UnboundLocalError if you > don't declare it nonlocal. >>>> >>>def f(): > eggs += lay_eggs() >>>> >>>f() > Traceback (most recent call last): > File "", line 1, in > f() > File "", line 2, in f > eggs += lay_eggs() > UnboundLocalError: local variable 'eggs' referenced before assignment > > Hence my previous statement that you need to write the function out of > line, which breaks the switch-ness of the block, and you definitely > need to declare eggs nonlocal. Now, if eggs is a list, you can extend > it with a method (which can be called in an expression), but if your > switch block can only do expressions, it's pretty limited. Another difference is a typical switch hash map is usually built at compile time in other languages. In python, the example's dict would be built every time it executes. That can be an advantage at times, but it can also make it slower than an if-elif's Moving the dict (and function definitions) to a place where it's only initialised once, may mean the names it access may not be reachable, so in most cases any values it depends on need to be passed as well. Ron From stephen at xemacs.org Sat Apr 26 17:07:56 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 27 Apr 2014 00:07:56 +0900 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: <87lhusnn77.fsf@uwakimon.sk.tsukuba.ac.jp> "Philipp A." wrote: > what y?all are missing about switch/case is that it?s *not* > necessarily a series of comparisons in most languages. I don't think anybody missed this. It's certainly been mentioned several times. Of course looking at a specific, full-featured example implementing the idea of "switch on a value" using (and abusing, as pointed out by Chris) existing Python features leads to the conclusion that such an implementation > is of course completely ugly and unnecessarily complicated for > cases where multiple keys lead to the same branch. I can't imagine anybody has missed this. It is presumably one reason why the OP is requesting new syntax. This also has a more fundamental flaw than "ugly" in many eyes: the need to define behavior *before* defining the case. The reason why people keep coming back to a series of comparisons (actually more general: a series of branch on tests) is that a series of comparisons is behaviorally equivalent to a switch. Written in appropriate style it's obvious that it is intended to have the effect of a switch. The code implementing a case is given "naturally" after mentioning the case. Given all that, it seems unnecessary to add new syntax. However, it's been mentioned that *if* we're going to have a case statement, it would be nice to give it "pattern matching" abilities such as the globbing of sh's 'case' or Haskell's signature matching. That power might be enough to get some people to swing over to supporting a syntax addition, it seems. Starting from that summary of the thread, what are you trying to argue? From steve at pearwood.info Sat Apr 26 17:17:23 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 27 Apr 2014 01:17:23 +1000 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: <20140426151723.GI4273@ando> On Sat, Apr 26, 2014 at 03:03:32PM +0200, Philipp A. wrote: > what y?all are missing about switch/case is that it?s *not* necessarily a > series of comparisons in most languages. *Some* people might be missing that, but not *all* of us :-) Before commenting further, anyone who isn't familiar with the two rejected PEPs on this topic should read them: http://legacy.python.org/dev/peps/pep-0275/ http://legacy.python.org/dev/peps/pep-3103/ Also, folks might like to review the wide range of semantics and syntax for switch/case in different languages: http://rosettacode.org/wiki/Conditional_Structures By my count, there are: 50 languages with no case/switch at all 30 languages using the keyword "case" 19 using "switch" 15 using "select" 3 using "when" 2 using "match" 2 using "given" (treating Perl 5 and 6 as distinct languages) 1 using "evaluate" 1 using "caseof" 1 using "if" (with a switch-like syntax) The above counts should be mostly accurate, but I don't guarantee that they are exactly correct. Apart from the possibility of counting errors, there were a few cases where I had to make a judgement call on whether or not language feature X was analogous to a switch or not. -- Steven From flying-sheep at web.de Sat Apr 26 19:44:23 2014 From: flying-sheep at web.de (Philipp A.) Date: Sat, 26 Apr 2014 19:44:23 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <87lhusnn77.fsf@uwakimon.sk.tsukuba.ac.jp> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> <87lhusnn77.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: 2014-04-26 17:07 GMT+02:00 Stephen J. Turnbull : > The reason why people keep coming back to a series of comparisons > (actually more general: a series of branch on tests) is that a series > of comparisons is behaviorally equivalent to a switch. Written in > appropriate style it's obvious that it is intended to have the effect > of a switch. The code implementing a case is given "naturally" after > mentioning the case. Given all that, it seems unnecessary to add new > syntax. > > Starting from that summary of the thread, what are you trying to > argue? > only that all the discussion about series of comparisons don?t paint the full picture; that the Case class isn?t enough. -------------- next part -------------- An HTML attachment was scrubbed... URL: From flying-sheep at web.de Sat Apr 26 19:52:29 2014 From: flying-sheep at web.de (Philipp A.) Date: Sat, 26 Apr 2014 19:52:29 +0200 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <20140426151723.GI4273@ando> References: <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> <20140426151723.GI4273@ando> Message-ID: 2014-04-26 17:17 GMT+02:00 Steven D'Aprano : > Also, folks might like to review the wide range of semantics and syntax > for switch/case in different languages: > > http://rosettacode.org/wiki/Conditional_Structures > you?re missing the language calling this ?pattern matching?: http://rosettacode.org/wiki/Pattern_matching i especially like scala?s way of doing it (even while recognizing that it?s no good fit for python) -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Sat Apr 26 19:59:55 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 26 Apr 2014 18:59:55 +0100 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <22741-1397762051-908018@sneakemail.com> References: <22741-1397762051-908018@sneakemail.com> Message-ID: On 17/04/2014 20:14, Lucas Malor wrote: [snipped] There are any one of five or six recipes on the web that are adequate for my needs, I've never seen a proposal that I really like and Python has survived for 23 years without one. Can we therefore shelve the switch-case debate until, say, Py5k? Yes, I meant 5 :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From tjreedy at udel.edu Sat Apr 26 21:38:01 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 26 Apr 2014 15:38:01 -0400 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> Message-ID: On 4/26/2014 10:44 AM, Philipp A. wrote: > interesting. it?s still assignment, even if nothing gets assigned (and > only |__iadd__| gets called behind the scenes). Augmented assignment statements are specialized assignment statements. They are documented in a subsection of the assignment statement section. https://docs.python.org/3/reference/simple_stmts.html#augmented-assignment-statements Something is always rebound, even if it is the same object. The interpreter does not know whether __iadd__ will return the same object or a new object -- and it does not check after. Not understanding that augmented assignment always assigns trips up beginners who mistakenly and unnecessarily try to use it to mutate a member of a tuple. >>> t = ([],) >>> t[0] += [1,2] Traceback (most recent call last): File "", line 1, in t[0] += [1,2] TypeError: 'tuple' object does not support item assignment >>> t ([1, 2],) This is equivalent to* and effectively executed as >>> t = ([],) >>> t[0] = t[0].__iadd__ ([1,2]) Traceback (most recent call last): File "", line 1, in t[0] = t[0].__iadd__ ([1,2]) TypeError: 'tuple' object does not support item assignment >>> t ([1, 2],) * The difference is that 't' is only evaluated once and the resulting reference is used for both subscriptions. The proper way to mutate a tuple member is to directly mutate it. >>> t = ([],) >>> t[0].extend([1,2]) >>> t ([1, 2],) -- Terry Jan Reedy From tjreedy at udel.edu Sat Apr 26 22:45:35 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 26 Apr 2014 16:45:35 -0400 Subject: [Python-ideas] Yet Another Switch-Case Syntax Proposal In-Reply-To: <87lhusnn77.fsf@uwakimon.sk.tsukuba.ac.jp> References: <22741-1397762051-908018@sneakemail.com> <4588-1397832212-797003@sneakemail.com> <5941-1397862578-751436@sneakemail.com> <0131291E-12B3-4E1C-B494-814452854143@yahoo.com> <20140423083803.GA7418@phconnel-ws.cisco.com> <87lhusnn77.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 4/26/2014 11:07 AM, Stephen J. Turnbull wrote: > However, it's been mentioned that *if* we're going to have a case > statement, it would be nice to give it "pattern matching" abilities > such as the globbing of sh's 'case' or Haskell's signature matching. > That power might be enough to get some people to swing over to > supporting a syntax addition, it seems. In my view, we should only add new syntax if it add substantial expressives -- making it easy to say something that is difficult now -- and not just be an alternative to if-elif-else. So I am one of 'some people'. -- Terry Jan Reedy