From ron3200 at gmail.com Wed Apr 1 07:11:08 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 01 Apr 2015 01:11:08 -0400 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com> Message-ID: On 03/30/2015 04:54 PM, Andrew Barnert wrote: > [snip] > >>> And the reason for bringing this idea up here was I think it could >>> be used to implement the lite macro behaviour that was suggested >>> with a bit of added syntax. >>> >>> On the other hand it appears to me, that Python is going in the >>> direction of making it easier to compile to C code. More dynamic >>> features may not be helpful in the long run. > I don't think this is true. There definitely isn't any momentum in that > direction--shedskin and its two competitors are dead, while PyPy and > numba are alive and kicking ass. And I don't think it's desirable, or > that the core developers think it's desirable. Good to know. BTW... here is a first try. Works barely. Don't use the decorators inside a function yet. I'm sure there's quite a few byte code changes that need to be done to make this dependable, but it does show it's possible and is something interesting to play with. ;-) This is closely related to continuations, but rather than try to capture a frame and then continue the code, it just applies code objects to a name space. That in it self ins't new, but getting the parts from decorated functions is a nice way to do it. import inspect def fix_code(c): """ Fix code object. This attempts to make a code object from a function be more like what you would get if you gave exec a string. * There is more that needs to be done to make this work dependably. But it's a start. """ varnames = c.co_varnames names = c.co_names xchg = [(124, 0x65), #LOAD_FAST to LOAD_NAME (125, 0x5a)] #STORE_FAST to STORE_NAME bcode = [] bgen = iter(c.co_code) for b in bgen: for bx1, bx2 in xchg: if b == bx1: i1 = next(bgen) i2 = next(bgen) index = i1 + i2 * 256 if b in [124, 125]: b = bx2 char = varnames[index] names = names + (char,) index = names.index(char) i2 = index // 256 i1 = index - i2 bcode.append(b) bcode.append(i1) bcode.append(i2) break else: bcode.append(b) co_code = bytes(bcode) Code = type(c) co = Code( 0, #co_argcount, 0, #co_kwonlyargcount, 0, #co_nlocals, c.co_stacksize, 64, #co_flags, co_code, c.co_consts, names, #co_names (), #co_varnames, c.co_filename, c.co_name, c.co_firstlineno, c.co_lnotab ) return co class fn_signature: """ Hold a signature from a function. When called with argumetns it will bind them and return a mapping. """ def __init__(self, fn): self.sig = inspect.signature(fn) def __call__(self, *args, **kwds): return dict(self.sig.bind(*args, **kwds).arguments) class fn_code: """ Create a relocatable code object that can be applied to a name space. """ def __init__(self, fn): self.co = fix_code(fn.__code__) def __call__(self, ns): return eval(self.co, ns) def get_sig(fn): """ Decorator to get a signature. """ return fn_signature(fn) def get_code(fn): """ Decorator to get a code object. """ return fn_code(fn) # Example 1 # Applying code to a namespace created by a signature. @get_sig def foo(x): pass @get_code def inc_x(): x += 1 @get_code def dec_x(): x -= 1 @get_code def get_x(): return x ns = foo(3) # creates a namespace inc_x(ns) # Apply code to namespace inc_x(ns) print(get_x(ns)) # --> 5 dec_x(ns) dec_x(ns) print(get_x(ns)) # --> 3 # Example 2 # Code + signature <---> function def add_xy(x, y): return x + y sig = get_sig(add_xy) co = get_code(add_xy) print(co(sig(3, 5))) # --> 8 From storchaka at gmail.com Wed Apr 1 14:28:39 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 01 Apr 2015 15:28:39 +0300 Subject: [Python-ideas] Clear all caches Message-ID: There are a lot of implicit caches in the stdlib. Some of them can grow unlimitedly. It would be good to have an official way to clear caches. I proposed to add two functions (in some existing lightweight module or add a new module): clear_caches(level=0) register_cache(level, clear_func) clear_caches() calls cache clearing functions for specified level and larger. register_cache() registers cache clearing functions for specified level. All modules that use implicit cache should register clearing function. functools.lru_cache() should register wrapper's cache clearing function. May be register gc.collect too. Third-party libraries and user code will be able to register their own clearing functions. From p.f.moore at gmail.com Wed Apr 1 15:27:54 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Apr 2015 14:27:54 +0100 Subject: [Python-ideas] Clear all caches In-Reply-To: References: Message-ID: On 1 April 2015 at 13:28, Serhiy Storchaka wrote: > I proposed to add two functions (in some existing lightweight module or add > a new module): > > clear_caches(level=0) > register_cache(level, clear_func) > > clear_caches() calls cache clearing functions for specified level and > larger. I'm not sure I understand how "level" would be used. Presumably anything registering a cache has to decide what level it wants to be at, but how should it decide? Maybe there should be some standard levels defined? Is there actually a need for a level parameter at all? Paul From storchaka at gmail.com Wed Apr 1 15:41:20 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 01 Apr 2015 16:41:20 +0300 Subject: [Python-ideas] Clear all caches In-Reply-To: References: Message-ID: On 01.04.15 16:27, Paul Moore wrote: > On 1 April 2015 at 13:28, Serhiy Storchaka wrote: >> I proposed to add two functions (in some existing lightweight module or add >> a new module): >> >> clear_caches(level=0) >> register_cache(level, clear_func) >> >> clear_caches() calls cache clearing functions for specified level and >> larger. > > I'm not sure I understand how "level" would be used. Presumably > anything registering a cache has to decide what level it wants to be > at, but how should it decide? Maybe there should be some standard > levels defined? Is there actually a need for a level parameter at all? All this can be discussed here. It is just an idea, I'm not sure about any details. Right now the level parameter is not needed to me, but may be other developers will need it. May be the behavior should be opposite: clear caches of specified level and *lower*. Then by default only level 0 will be cleared and users will be able to use higher levels for long-living caches. From abarnert at yahoo.com Wed Apr 1 16:25:17 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 07:25:17 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com> Message-ID: <630B1380-73C5-4FA2-B67A-3DC2786F08A4@yahoo.com> On Mar 31, 2015, at 22:11, Ron Adam wrote: > > > >> On 03/30/2015 04:54 PM, Andrew Barnert wrote: >> [snip] >> >>>> And the reason for bringing this idea up here was I think it could >>>> be used to implement the lite macro behaviour that was suggested >>>> with a bit of added syntax. >>>> >>>> On the other hand it appears to me, that Python is going in the >>>> direction of making it easier to compile to C code. More dynamic >>>> features may not be helpful in the long run. > >> I don't think this is true. There definitely isn't any momentum in that >> direction--shedskin and its two competitors are dead, while PyPy and >> numba are alive and kicking ass. And I don't think it's desirable, or >> that the core developers think it's desirable. > > Good to know. > > BTW... here is a first try. Works barely. Don't use the decorators inside a function yet. I'm sure there's quite a few byte code changes that need to be done to make this dependable, but it does show it's possible and is something interesting to play with. ;-) > > This is closely related to continuations, but rather than try to capture a frame and then continue the code, it just applies code objects to a name space. That in it self ins't new, but getting the parts from decorated functions is a nice way to do it. > > > import inspect > > def fix_code(c): > """ > Fix code object. > > This attempts to make a code object from a function be > more like what you would get if you gave exec a string. > > * There is more that needs to be done to > make this work dependably. But it's a start. > > """ > varnames = c.co_varnames > names = c.co_names > > xchg = [(124, 0x65), #LOAD_FAST to LOAD_NAME > (125, 0x5a)] #STORE_FAST to STORE_NAME The first problem here is that any 124 or 125 in an operand to any opcode except 124 or 125 will be matched and converted (although you'll usually probably get an IndexError trying to treat the next two arbitrary bytes as an index...). To solve this, you need to iterate opcode by opcode, not byte by byte. The dis module gives you the information to tell how many bytes to skip for each opcode's operands. (It also maps between opcode numbers and names, so you don't have to use magic numbers with comments.) Using it will solve this problem (and maybe others I didn't spot) and also make your code a lot simpler. Another problem is that this will only catch local variables. Anything you don't assign to in the function but do reference is liable to end up a global or cell, which will still be a global or cell when you try to run it later. I'm not sure exactly how you want to handle these (if you just convert them unconditionally, then it'll be a bit surprising if someone writes "global spam" and doesn't get a global...), but you have to do something, or half your motivating examples (like "lambda: x+1") won't work. (Or is that why you're doing the explicit namespace thing instead of using the actual scope later on, so that this won't break, because the explicit namespace is both your locals and your globals?) > bcode = [] > bgen = iter(c.co_code) > for b in bgen: > for bx1, bx2 in xchg: > if b == bx1: > i1 = next(bgen) > i2 = next(bgen) > index = i1 + i2 * 256 > if b in [124, 125]: > b = bx2 > char = varnames[index] > names = names + (char,) > index = names.index(char) > i2 = index // 256 > i1 = index - i2 > bcode.append(b) > bcode.append(i1) > bcode.append(i2) > break > else: > bcode.append(b) > co_code = bytes(bcode) > > Code = type(c) > co = Code( > 0, #co_argcount, > 0, #co_kwonlyargcount, > 0, #co_nlocals, > c.co_stacksize, > 64, #co_flags, > co_code, > c.co_consts, > names, #co_names > (), #co_varnames, > c.co_filename, > c.co_name, > c.co_firstlineno, > c.co_lnotab > ) > return co > > > class fn_signature: > """ > Hold a signature from a function. > > When called with argumetns it will bind them > and return a mapping. > """ > def __init__(self, fn): > self.sig = inspect.signature(fn) > > def __call__(self, *args, **kwds): > return dict(self.sig.bind(*args, **kwds).arguments) > > > class fn_code: > """ > Create a relocatable code object that can > be applied to a name space. > """ > def __init__(self, fn): > self.co = fix_code(fn.__code__) > > def __call__(self, ns): > return eval(self.co, ns) Why are you applying these to a dict? I thought the whole point was to be able to run it inside a scope and affect that scope's variables? If you just leave out the ns, won't that be closer to what you want? (And it also means you don't need the get_sig thing in the first place, and I'm not sure what that adds. Using a function signature plus a call expression as a fancy way of writing a dict display seems like just obfuscation. Maybe with default values, *args, binding partials and then calling get_sig on them, etc. is interesting for something, but I'm not sure what?) > def get_sig(fn): > """ Decorator to get a signature. """ > return fn_signature(fn) > > > def get_code(fn): > """ Decorator to get a code object. """ > return fn_code(fn) > > > > > > # Example 1 > # Applying code to a namespace created by a signature. > > @get_sig > def foo(x): > pass > > @get_code > def inc_x(): > x += 1 > > @get_code > def dec_x(): > x -= 1 > > @get_code > def get_x(): > return x > > > ns = foo(3) # creates a namespace > > inc_x(ns) # Apply code to namespace > inc_x(ns) > print(get_x(ns)) # --> 5 > > dec_x(ns) > dec_x(ns) > print(get_x(ns)) # --> 3 > > > > # Example 2 > # Code + signature <---> function > > def add_xy(x, y): > return x + y > > sig = get_sig(add_xy) > co = get_code(add_xy) > print(co(sig(3, 5))) # --> 8 > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Wed Apr 1 16:37:31 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 07:37:31 -0700 Subject: [Python-ideas] Clear all caches In-Reply-To: References: Message-ID: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> On Apr 1, 2015, at 06:41, Serhiy Storchaka wrote: > >> On 01.04.15 16:27, Paul Moore wrote: >>> On 1 April 2015 at 13:28, Serhiy Storchaka wrote: >>> I proposed to add two functions (in some existing lightweight module or add >>> a new module): >>> >>> clear_caches(level=0) >>> register_cache(level, clear_func) >>> >>> clear_caches() calls cache clearing functions for specified level and >>> larger. I can see this being useful as just a way of standardizing an API for third-party modules that have a cache-clearing function. If there are a lot of them, having most of them do it the same way would make them easier to discover. >> I'm not sure I understand how "level" would be used. Presumably >> anything registering a cache has to decide what level it wants to be >> at, but how should it decide? Maybe there should be some standard >> levels defined? Is there actually a need for a level parameter at all? > > All this can be discussed here. It is just an idea, I'm not sure about any details. Right now the level parameter is not needed to me, but may be other developers will need it. > > May be the behavior should be opposite: clear caches of specified level and *lower*. Then by default only level 0 will be cleared and users will be able to use higher levels for long-living caches. Higher vs. lower doesn't really matter that much; you can just define short, medium, and long as 0, -1, -2 instead of 0, 1, 2, right? (I think that, even if you don't define standard levels by name as Paul Moore suggests, you're still going to be doing so implicitly by the levels you choose in the stdlib...) Anyway, why do you want this? Is there some cache that's using too much memory in an app of yours? Or is it more about getting a semi-clean start on the interactive interpreter? Or something different? And which of the implicit caches all over the stdlib made you want this (since the stdlib was your motivating example)? Also, this seems like something that other platforms (special-purpose math languages, interactive SQL interpreters, etc.) might have. If so, have you looked around to see how they do it? From szport at gmail.com Wed Apr 1 15:55:50 2015 From: szport at gmail.com (Zaur Shibzukhov) Date: Wed, 1 Apr 2015 06:55:50 -0700 (PDT) Subject: [Python-ideas] recordclass: a mutable alternative to namedtuple In-Reply-To: <20150328133709.GE25453@ando.pearwood.info> References: <6b53b718-0220-48d5-b5b5-ded2fa9e867c@googlegroups.com> <0CF24FB3-2E88-4A44-840F-358A4498B4EF@yahoo.com> <20150328133709.GE25453@ando.pearwood.info> Message-ID: <34477d2f-70f3-4e10-9545-4ddb651c0af1@googlegroups.com> It is possible that this is recordclass A short example to illustrate that fact. ???????, 28 ????? 2015 ?., 16:37:40 UTC+3 ???????????? Steven D'Aprano ???????: > > On Fri, Mar 27, 2015 at 04:13:46PM -0700, Andrew Barnert wrote: > > On Mar 27, 2015, at 06:22, Joao S. O. Bueno > wrote: > [...] > > > The Python equivalent of a C Struct. > > > > But a C struct is not a subtype of, or substitutable for, a C array. > > It's not indexable. And the same is true with the equivalents in other > > languages. In fact, the dichotomy between struct--heterogeneous > > collection of fixed named fields--and array--homogeneous collection of > > indexed fields--goes back way before C. So, if you want the equivalent > > of a C struct, there's no reason to make it an iterable in Python. > > Joao said "The Python equivalent of a C struct", not "a C struct". > Python is not C, and Python data types are not limited to what C does. > Python strings aren't limited to C null-delimited strings, and Python > ints aren't limited to what C ints can do. > > I think the original thread was quite explicit about what is wanted: > something like a mutable equivalent to namedtuple. Namedtuples are used > in Python where C would use a struct, or Pascal a record, except that > namedtuples (being tuples) are immutable. I think it's quite reasonable > to want a mutable version. > > Effectively, namedtuple is just a convenience function for wrapping up a > bunch of nice-to-have but not essential functionality around an > immutable struct. Python got by with unnamed tuples for over a decade, > so it's not like we *have* to have namedtuples. But having got them, > would we go back to using regular tuples as a struct? Hell no. Having > named fields is so much better. > > > > > And a class already is the Python of a C struct, it's just that it can > > do _more_ than a C struct. > > This is why it is unfair to insist that a Python equivalent of a C > struct be limited to what C structs do. > > > > > Just that. > > > An easy to create class, with named fields, > > > > Which is easy to do: just create a class, and create its fields in the > > __init__ method (or, in some cases, it's acceptable to use class > > attributes as "default values" for instance attributes). > > If this is so easy, why we have namedtuple *and* SimpleNamespace > in the standard library. Are they both mistakes? > > SimpleNamespace is especially interesting. The docs say: > > "However, for a structured record type use namedtuple() instead." > > https://docs.python.org/3/library/types.html#types.SimpleNamespace > > > which is great if you want an *immutable* structured record type, but > not if you want a mutable one. > > Which brings us back to where this thread started: a request for a > mutable version of namedtuple. That's trickier than namedtuple, because > we don't have a mutable version of a tuple to inherit from. Lists won't > do the job, because they have a whole lot of functionality that are > inappropriate, e.g. sort, reverse, pop methods. > > That makes it harder to create a mutable structured record type, not > simpler. > > Think about the functional requirements: > > - it should be semantically a struct, not a list or array; > > - with a fixed set of named fields; > > - fields should be ordered: a record with fields foo and bar is not the > same as a record with fields bar and foo; > > - accessing fields by index would be a Nice To Have, but not essential; > > - but iteration is essential, for sequence unpacking; > > - values in the fields must be mutable; > > - it should support equality, but not hashing (since it is mutable); > > - it must have a nice repr and/or str; > > - being mutable, it may directly or indirectly contain a reference to > itself (e.g. x.field = x) so it needs to deal with that correctly; > > - support for pickle; > > - like namedtuple, it may benefit from a handful of methods such as > '_asdict', '_fields', '_make', '_replace' or similar. > > > Does this sound easy to write? Well, sure, in the big picture, it's > hardly a 100,000 line application. But it's not a trivial class. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Wed Apr 1 16:58:44 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Wed, 01 Apr 2015 10:58:44 -0400 Subject: [Python-ideas] Clear all caches In-Reply-To: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> Message-ID: <551C07A4.6050303@sdamon.com> Why not just take a dbapi approach to this; define an api, build things to that api in the standard library, but leave it to third parties to use it. "Does your module have a cache? define these methods somewhere, and anything that manages cache will know how to deal with it". No global registry, not global manager in the standard library, just an api. On 4/1/2015 10:37, Andrew Barnert wrote: > On Apr 1, 2015, at 06:41, Serhiy Storchaka wrote: >>> On 01.04.15 16:27, Paul Moore wrote: >>>> On 1 April 2015 at 13:28, Serhiy Storchaka wrote: >>>> I proposed to add two functions (in some existing lightweight module or add >>>> a new module): >>>> >>>> clear_caches(level=0) >>>> register_cache(level, clear_func) >>>> >>>> clear_caches() calls cache clearing functions for specified level and >>>> larger. > I can see this being useful as just a way of standardizing an API for third-party modules that have a cache-clearing function. If there are a lot of them, having most of them do it the same way would make them easier to discover. > >>> I'm not sure I understand how "level" would be used. Presumably >>> anything registering a cache has to decide what level it wants to be >>> at, but how should it decide? Maybe there should be some standard >>> levels defined? Is there actually a need for a level parameter at all? >> All this can be discussed here. It is just an idea, I'm not sure about any details. Right now the level parameter is not needed to me, but may be other developers will need it. >> >> May be the behavior should be opposite: clear caches of specified level and *lower*. Then by default only level 0 will be cleared and users will be able to use higher levels for long-living caches. > Higher vs. lower doesn't really matter that much; you can just define short, medium, and long as 0, -1, -2 instead of 0, 1, 2, right? (I think that, even if you don't define standard levels by name as Paul Moore suggests, you're still going to be doing so implicitly by the levels you choose in the stdlib...) > > Anyway, why do you want this? Is there some cache that's using too much memory in an app of yours? Or is it more about getting a semi-clean start on the interactive interpreter? Or something different? > > And which of the implicit caches all over the stdlib made you want this (since the stdlib was your motivating example)? > > Also, this seems like something that other platforms (special-purpose math languages, interactive SQL interpreters, etc.) might have. If so, have you looked around to see how they do it? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Wed Apr 1 17:03:24 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 1 Apr 2015 08:03:24 -0700 Subject: [Python-ideas] Clear all caches In-Reply-To: <551C07A4.6050303@sdamon.com> References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <551C07A4.6050303@sdamon.com> Message-ID: The elephant in the room is: what are those caches? I'd suggest starting without the level. On Apr 1, 2015 7:59 AM, "Alexander Walters" wrote: > Why not just take a dbapi approach to this; define an api, build things to > that api in the standard library, but leave it to third parties to use it. > "Does your module have a cache? define these methods somewhere, and > anything that manages cache will know how to deal with it". No global > registry, not global manager in the standard library, just an api. > > On 4/1/2015 10:37, Andrew Barnert wrote: > >> On Apr 1, 2015, at 06:41, Serhiy Storchaka wrote: >> >>> On 01.04.15 16:27, Paul Moore wrote: >>>> >>>>> On 1 April 2015 at 13:28, Serhiy Storchaka >>>>> wrote: >>>>> I proposed to add two functions (in some existing lightweight module >>>>> or add >>>>> a new module): >>>>> >>>>> clear_caches(level=0) >>>>> register_cache(level, clear_func) >>>>> >>>>> clear_caches() calls cache clearing functions for specified level and >>>>> larger. >>>>> >>>> I can see this being useful as just a way of standardizing an API for >> third-party modules that have a cache-clearing function. If there are a lot >> of them, having most of them do it the same way would make them easier to >> discover. >> >> I'm not sure I understand how "level" would be used. Presumably >>>> anything registering a cache has to decide what level it wants to be >>>> at, but how should it decide? Maybe there should be some standard >>>> levels defined? Is there actually a need for a level parameter at all? >>>> >>> All this can be discussed here. It is just an idea, I'm not sure about >>> any details. Right now the level parameter is not needed to me, but may be >>> other developers will need it. >>> >>> May be the behavior should be opposite: clear caches of specified level >>> and *lower*. Then by default only level 0 will be cleared and users will be >>> able to use higher levels for long-living caches. >>> >> Higher vs. lower doesn't really matter that much; you can just define >> short, medium, and long as 0, -1, -2 instead of 0, 1, 2, right? (I think >> that, even if you don't define standard levels by name as Paul Moore >> suggests, you're still going to be doing so implicitly by the levels you >> choose in the stdlib...) >> >> Anyway, why do you want this? Is there some cache that's using too much >> memory in an app of yours? Or is it more about getting a semi-clean start >> on the interactive interpreter? Or something different? >> >> And which of the implicit caches all over the stdlib made you want this >> (since the stdlib was your motivating example)? >> >> Also, this seems like something that other platforms (special-purpose >> math languages, interactive SQL interpreters, etc.) might have. If so, have >> you looked around to see how they do it? >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Apr 1 17:17:16 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 1 Apr 2015 09:17:16 -0600 Subject: [Python-ideas] Clear all caches In-Reply-To: <551C07A4.6050303@sdamon.com> References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <551C07A4.6050303@sdamon.com> Message-ID: On Wed, Apr 1, 2015 at 8:58 AM, Alexander Walters wrote: > Why not just take a dbapi approach to this; define an api, build things to > that api in the standard library, but leave it to third parties to use it. > "Does your module have a cache? define these methods somewhere, and > anything that manages cache will know how to deal with it". No global > registry, not global manager in the standard library, just an api. What Serhiy is proposing is a mechanism for clearing a bunch of caches at once. So you need a registry of caches that are included in that operation. With just a protocol, Serhiy's goal would have to be accomplished by somehow scanning through all live objects for those that implement. An explicit opt-in mechanism is both more efficient and friendlier. -eric From ericsnowcurrently at gmail.com Wed Apr 1 17:18:19 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 1 Apr 2015 09:18:19 -0600 Subject: [Python-ideas] Clear all caches In-Reply-To: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> Message-ID: On Wed, Apr 1, 2015 at 8:37 AM, Andrew Barnert wrote: > Anyway, why do you want this? Is there some cache that's using too much memory in an app of yours? Or is it more about getting a semi-clean start on the interactive interpreter? Or something different? Presumably http://bugs.python.org/issue23839. -eric From p.f.moore at gmail.com Wed Apr 1 17:27:31 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Apr 2015 16:27:31 +0100 Subject: [Python-ideas] Clear all caches In-Reply-To: References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <551C07A4.6050303@sdamon.com> Message-ID: On 1 April 2015 at 16:03, Guido van Rossum wrote: > The elephant in the room is: what are those caches? A quick scan of the regrtest code mentioned in http://bugs.python.org/issue23839 suggest that the following would be a start: copyreg dispatch tale sys.path_importer_cache zipimport._zip_directory_cache ABC registry cache(s) sys._clear_type_cache() distutils._dir_util._path_created.clear() re.purge() _strptime._regex_cache.clear() urllib.parse.clear_cache() urllib.request.urlcleanup() linecache.clearcache() mimetypes._default_mime_types() filecmp._cache.clear() struct._clearcache() ctypes._reset_cache() That's actually quite a lot of caches... Paul From skip.montanaro at gmail.com Wed Apr 1 17:29:01 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 1 Apr 2015 10:29:01 -0500 Subject: [Python-ideas] Clear all caches In-Reply-To: References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <551C07A4.6050303@sdamon.com> Message-ID: The other elephant might be... If your caches can grow without bound so that programmers need to know about them (so they can clear them, in testing or other situations), then maybe they aren't well-designed. If cache growth is an issue, I'd rather see an abstract API for LRU (and other) cache types which intelligently cap cache size. I suspect that long before you hit the inevitable MemoryError, unbounded caches probably destroy program performance by causing large-scale swapping. Skip From storchaka at gmail.com Wed Apr 1 17:32:28 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 01 Apr 2015 18:32:28 +0300 Subject: [Python-ideas] Clear all caches In-Reply-To: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> Message-ID: On 01.04.15 17:37, Andrew Barnert wrote: > I can see this being useful as just a way of standardizing an API for > third-party modules that have a cache-clearing function. If there are > a lot of them, having most of them do it the same way would make them > easier to discover. How will you discover them? Do you want to enumerate all loaded modules and all functions in these modules to check if they look as cache clearing functions? What about caches in class scope and dynamically created caches? >> May be the behavior should be opposite: clear caches of specified >> level and *lower*. Then by default only level 0 will be cleared and >> users will be able to use higher levels for long-living caches. > > Higher vs. lower doesn't really matter that much; you can just define > short, medium, and long as 0, -1, -2 instead of 0, 1, 2, right? Right. But 0, 1, 2 can look less strange than 0, -1, -2. If all actually used values are negative, then we should revert the scale. > Anyway, why do you want this? Is there some cache that's using too > much memory in an app of yours? Or is it more about getting a > semi-clean start on the interactive interpreter? Or something > different? This idea was inspired by a series of MemoryErrors on some buildbots. Some of these errors are provoked by overfilled caches (in particular linecache). I don't know how large other caches can be, but linecache can grow over 23 MB only on the stdlib and standard tests. See also issues #23838 and #23839. http://bugs.python.org/issue23838 http://bugs.python.org/issue23839 > And which of the implicit caches all over the stdlib made you want > this (since the stdlib was your motivating example)? First of all it's linecache. But some cache can grow even more. For example re cache grows unlimitedly if you generates patterns dynamically from user data and use module level functions. > Also, this seems like something that other platforms (special-purpose > math languages, interactive SQL interpreters, etc.) might have. If > so, have you looked around to see how they do it? I don't see how this idea implemented in particular library can help to solve general problem without support in the stdlib. From abarnert at yahoo.com Wed Apr 1 17:32:41 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 08:32:41 -0700 Subject: [Python-ideas] Clear all caches In-Reply-To: References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> Message-ID: <40A66F6E-99FE-44D9-8EE4-4ABF8152D6D7@yahoo.com> On Apr 1, 2015, at 08:18, Eric Snow wrote: > > On Wed, Apr 1, 2015 at 8:37 AM, Andrew Barnert > wrote: >> Anyway, why do you want this? Is there some cache that's using too much memory in an app of yours? Or is it more about getting a semi-clean start on the interactive interpreter? Or something different? > > Presumably http://bugs.python.org/issue23839. Ah, for running a large test suite (like the stdlib's). That makes sense. From the initial report ("in particular linecache") it seems at least possible that there's really just one cache that's a real issue, in which case there's a pretty obvious fix. (And making it possible to use linecache in a bounded way, instead of all-or-nothing clearing, might be useful for other purposes besides this...) But I don't know if that really is the case; did anyone check whether the 500MB were 90%+ in linecache or anything like that? From guido at python.org Wed Apr 1 17:35:32 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 1 Apr 2015 08:35:32 -0700 Subject: [Python-ideas] Clear all caches In-Reply-To: References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <551C07A4.6050303@sdamon.com> Message-ID: Actually several of these "caches" aren't caches at all -- deleting data from them will break APIs. This is true at least for copyreg, the ABC registry (though the things with cache in their names in ABC *are* proper caches), and the mime types registry. Also urllib.request.urlcleanup() removes the opener set by the install_opener() API. On Wed, Apr 1, 2015 at 8:29 AM, Skip Montanaro wrote: > The other elephant might be... > > If your caches can grow without bound so that programmers need to know > about them (so they can clear them, in testing or other situations), > then maybe they aren't well-designed. If cache growth is an issue, I'd > rather see an abstract API for LRU (and other) cache types which > intelligently cap cache size. I suspect that long before you hit the > inevitable MemoryError, unbounded caches probably destroy program > performance by causing large-scale swapping. > > Skip > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Wed Apr 1 17:45:52 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 1 Apr 2015 10:45:52 -0500 Subject: [Python-ideas] Clear all caches In-Reply-To: References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <551C07A4.6050303@sdamon.com> Message-ID: On Wed, Apr 1, 2015 at 10:35 AM, Guido van Rossum wrote: > Actually several of these "caches" aren't caches at all -- deleting data > from them will break APIs. Then, like you say, they aren't caches. If we split them into two groups (fake and real), then the real caches (like linecache) would probably benefit with some actual cache-like trimming. The others should be fixed in some other way. Will these problems only be evident during testing, or are there plausible scenarios where a real application could stress the system? Or could such unbounded registries/caches be potential vectors for DOS attacks? A cache API still makes some sense to me, even if only to have available for application writers. I see that functools has an lru_cache decorator. Perhaps linecache could use it? Skip From guido at python.org Wed Apr 1 17:50:00 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 1 Apr 2015 08:50:00 -0700 Subject: [Python-ideas] Clear all caches In-Reply-To: <40A66F6E-99FE-44D9-8EE4-4ABF8152D6D7@yahoo.com> References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <40A66F6E-99FE-44D9-8EE4-4ABF8152D6D7@yahoo.com> Message-ID: Regardless of whether we come up with a general solution (seems of doubtful use to me), linecache really is last-century technology. :-( On Wed, Apr 1, 2015 at 8:32 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Apr 1, 2015, at 08:18, Eric Snow wrote: > > > > On Wed, Apr 1, 2015 at 8:37 AM, Andrew Barnert > > wrote: > >> Anyway, why do you want this? Is there some cache that's using too much > memory in an app of yours? Or is it more about getting a semi-clean start > on the interactive interpreter? Or something different? > > > > Presumably http://bugs.python.org/issue23839. > > Ah, for running a large test suite (like the stdlib's). That makes sense. > > From the initial report ("in particular linecache") it seems at least > possible that there's really just one cache that's a real issue, in which > case there's a pretty obvious fix. (And making it possible to use linecache > in a bounded way, instead of all-or-nothing clearing, might be useful for > other purposes besides this...) But I don't know if that really is the > case; did anyone check whether the 500MB were 90%+ in linecache or anything > like that? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Apr 1 17:52:08 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 01 Apr 2015 18:52:08 +0300 Subject: [Python-ideas] Clear all caches In-Reply-To: References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <551C07A4.6050303@sdamon.com> Message-ID: On 01.04.15 18:03, Guido van Rossum wrote: > The elephant in the room is: what are those caches? More than a dozen of caches are mentioned in Lib/test/regrtest.py. Some of these caches even have not public interface. And perhaps this list is not complete. > I'd suggest starting without the level. Well, then API will be simpler. Then next question: where place these functions? The module should be lightweight, so it can be imported in any module with a cache without weighting it. From rymg19 at gmail.com Wed Apr 1 19:32:26 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 1 Apr 2015 12:32:26 -0500 Subject: [Python-ideas] A new module loader Message-ID: Attached is a new module loader I've been working on that's an order of magnitude times faster than the current one. What would it take for this to be merged? BTW... (rot13) Ncevy Sbbyf! -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: new_import.tar Type: application/x-tar Size: 10240 bytes Desc: not available URL: From rosuav at gmail.com Wed Apr 1 19:51:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 2 Apr 2015 04:51:37 +1100 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On Thu, Apr 2, 2015 at 4:32 AM, Ryan Gonzalez wrote: > Attached is a new module loader I've been working on that's an order of > magnitude times faster than the current one. What would it take for this to > be merged? LGTM. Before you can get this merged, you'll need to sign a contributor agreement, if you haven't already. Further discussion of the proposal's details can happen on the tracker: http://bugs.python.org/issue0401 ChrisA From rymg19 at gmail.com Wed Apr 1 19:52:24 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 1 Apr 2015 12:52:24 -0500 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: Did you open the archive? On Wed, Apr 1, 2015 at 12:51 PM, Chris Angelico wrote: > On Thu, Apr 2, 2015 at 4:32 AM, Ryan Gonzalez wrote: > > Attached is a new module loader I've been working on that's an order of > > magnitude times faster than the current one. What would it take for this > to > > be merged? > > LGTM. Before you can get this merged, you'll need to sign a > contributor agreement, if you haven't already. Further discussion of > the proposal's details can happen on the tracker: > > http://bugs.python.org/issue0401 > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Apr 1 19:56:50 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 2 Apr 2015 04:56:50 +1100 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On Thu, Apr 2, 2015 at 4:52 AM, Ryan Gonzalez wrote: > Did you open the archive? I did, and explored the code. It's so obviously an improvement over the current state that there's no reason to bikeshed the details. ChrisA From skip.montanaro at gmail.com Wed Apr 1 20:20:12 2015 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 1 Apr 2015 13:20:12 -0500 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: I like how it also tidies up the filesystem at the same time. Skip From ron3200 at gmail.com Wed Apr 1 21:39:49 2015 From: ron3200 at gmail.com (Ron Adam) Date: Wed, 01 Apr 2015 15:39:49 -0400 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <630B1380-73C5-4FA2-B67A-3DC2786F08A4@yahoo.com> References: <20150328172624.GG25453@ando.pearwood.info> <36A4F7D4-20A4-4BE9-9D86-B423C13A7320@yahoo.com> <20150329015144.GJ25453@ando.pearwood.info> <4C42926A-2DAE-4BB3-86E6-FC248C5C729D@yahoo.com> <710F1B95-B063-44C1-9BB4-E8AB9F1B9133@yahoo.com> <4E099DC0-B146-476E-833C-3CF73CD53B9A@yahoo.com> <630B1380-73C5-4FA2-B67A-3DC2786F08A4@yahoo.com> Message-ID: On 04/01/2015 10:25 AM, Andrew Barnert wrote: >> > xchg = [(124, 0x65), #LOAD_FAST to LOAD_NAME >> > (125, 0x5a)] #STORE_FAST to STORE_NAME > The first problem here is that any 124 or 125 in an operand to any > opcode except 124 or 125 will be matched and converted (although you'll usually > probably get an IndexError trying to treat the next two arbitrary bytes as > an index...). Yes, it will only work for very simple cases. It was just enough to get the initial examples working. > To solve this, you need to iterate opcode by opcode, not byte by byte. > The dis module gives you the information to tell how many bytes to skip for > each opcode's operands. (It also maps between opcode numbers and names, so > you don't have to use magic numbers with comments.) Using it will solve > this problem (and maybe others I didn't spot) and also make your code a lot > simpler. Unfortunately dis is written to give human output for python bytecode, not to edit bytecode. But it can help. It needs a function to go back to a code object after editing the instruction list. > Another problem is that this will only catch local variables. Anything > you don't assign to in the function but do reference is liable to end up a > global or cell, which will still be a global or cell when you try to run it > later. I'm not sure exactly how you want to handle these (if you just > convert them unconditionally, then it'll be a bit surprising if someone > writes "global spam" and doesn't get a global...), but you have to do > something, or half your motivating examples (like "lambda: x+1") won't > work. (Or is that why you're doing the explicit namespace thing instead of > using the actual scope later on, so that this won't break, because the > explicit namespace is both your locals and your globals?) > Why are you applying these to a dict? I thought the whole point was to > be able to run it inside a scope and affect that scope's variables? If > you just leave out the ns, won't that be closer to what you want? (And > it also means you don't need the get_sig thing in the first place, and > I'm not sure what that adds. Using a function signature plus a call > expression as a fancy way of writing a dict display seems like just > obfuscation. Maybe with default values, *args, binding partials and then > calling get_sig on them, etc. is interesting for something, but I'm not > sure what?) This was just a step in that direction. It obviously needs more work. There are a number of interesting aspects and directions this can go. * Ability to decompose functions into separate signature and body parts. Not useful (or hard) in itself, but being able to reuse those parts may be good. * Use those parts together. This is a basic test that should just work. Again it's not really that useful in itself, but it's a nice to have equivalency and helps to demonstrate how it works. * Use one body with different signatures. For example you might have signatures with different default values, or signatures that interface to different data formats. Rather than having a function that converts different data formats to fit a single signature format, we can just use different signatures to interface with the data more directly. This is one of the things macro's do in other languages. * Use code objects as blocks to implement continuation like behaviours. This is done breaking an algorithm into composable parts, then applying them to data. It's not quite the same as continuations, or generators, but has some of the same benefits. If the blocks avoid parsing signatures and creating/destroying frames, it can be a fast way to translate data. Of course, it's very limited as you need to have strict naming conventions to do this. So it would be limited to within a scope that follows those conventions. (Doable now with compile and exec, but it's awkward in my opinion.) * Use a body as a block in another function. Yes, this requires getting the live namespace from the frame it's used in. f = sys_getframe(-1) may work, but it's not that easy to do in this case. When exec is given a code object, it call's PyEval_EvalCodeEx in ceval.c directly with local and global dictionaries. (That should answer some of your comments as to why I uses the dictionary.) It may be possible to call directly to PyEval_EvalFrameEx (like generators do), with a frame object. Some or all of these may/will require the code object to be in a form that is more easily relocatable, but as you noted, its not easy to do. There are a lot of 'ifs' here, but I think it may be worth exploring. I'm going to try and make the bytecode fixer function work better (using dis or parts of it.) And then put it up on github where this idea can be developed further. (The other utilities I've found so far for editing bytecode aren't ported to python3 yet.) I don't think editing the bytecode is the ideal solution in the long run, but it will identify the parts that need addressing and then other solutions for those could be looked into such as doing the needed alterations in the AST rather than the bytecode. Cheers, Ron From robertc at robertcollins.net Wed Apr 1 22:27:29 2015 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 2 Apr 2015 09:27:29 +1300 Subject: [Python-ideas] Clear all caches In-Reply-To: <40A66F6E-99FE-44D9-8EE4-4ABF8152D6D7@yahoo.com> References: <00E0E78A-DC3E-46AF-A103-96CD1EB097EA@yahoo.com> <40A66F6E-99FE-44D9-8EE4-4ABF8152D6D7@yahoo.com> Message-ID: On 2 April 2015 at 04:32, Andrew Barnert wrote: > On Apr 1, 2015, at 08:18, Eric Snow wrote: >> >> On Wed, Apr 1, 2015 at 8:37 AM, Andrew Barnert >> wrote: >>> Anyway, why do you want this? Is there some cache that's using too much memory in an app of yours? Or is it more about getting a semi-clean start on the interactive interpreter? Or something different? >> >> Presumably http://bugs.python.org/issue23839. > > Ah, for running a large test suite (like the stdlib's). That makes sense. > > From the initial report ("in particular linecache") it seems at least possible that there's really just one cache that's a real issue, in which case there's a pretty obvious fix. (And making it possible to use linecache in a bounded way, instead of all-or-nothing clearing, might be useful for other purposes besides this...) But I don't know if that really is the case; did anyone check whether the 500MB were 90%+ in linecache or anything like that? We can probably tune the new traceback code to avoid linecache getting populated at all on a passing testrun. That would be better than wiping it out on every test, which if its being hit will just trade memory for CPU - slower test runs. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From breamoreboy at yahoo.co.uk Wed Apr 1 23:05:44 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 01 Apr 2015 22:05:44 +0100 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On 01/04/2015 18:51, Chris Angelico wrote: > On Thu, Apr 2, 2015 at 4:32 AM, Ryan Gonzalez wrote: >> Attached is a new module loader I've been working on that's an order of >> magnitude times faster than the current one. What would it take for this to >> be merged? > > LGTM. Before you can get this merged, you'll need to sign a > contributor agreement, if you haven't already. Further discussion of > the proposal's details can happen on the tracker: > > http://bugs.python.org/issue0401 > > ChrisA What's the actual issue number, I get "There is no issue with id 401" ? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From storchaka at gmail.com Wed Apr 1 23:14:28 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 02 Apr 2015 00:14:28 +0300 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On 02.04.15 00:05, Mark Lawrence wrote: > On 01/04/2015 18:51, Chris Angelico wrote: >> On Thu, Apr 2, 2015 at 4:32 AM, Ryan Gonzalez >> wrote: >>> Attached is a new module loader I've been working on that's an order of >>> magnitude times faster than the current one. What would it take for >>> this to >>> be merged? >> >> LGTM. Before you can get this merged, you'll need to sign a >> contributor agreement, if you haven't already. Further discussion of >> the proposal's details can happen on the tracker: >> >> http://bugs.python.org/issue0401 >> >> ChrisA > > What's the actual issue number, I get "There is no issue with id 401" ? Cris missed. The correct link is http://bugs.python.org/issue0104. From jeanpierreda at gmail.com Wed Apr 1 23:14:49 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 1 Apr 2015 14:14:49 -0700 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: I think Chris meant to link to http://bugs.python.org/issue0404 , sorry for any confusion. -- Devin On Wed, Apr 1, 2015 at 2:05 PM, Mark Lawrence wrote: > On 01/04/2015 18:51, Chris Angelico wrote: >> >> On Thu, Apr 2, 2015 at 4:32 AM, Ryan Gonzalez wrote: >>> >>> Attached is a new module loader I've been working on that's an order of >>> magnitude times faster than the current one. What would it take for this >>> to >>> be merged? >> >> >> LGTM. Before you can get this merged, you'll need to sign a >> contributor agreement, if you haven't already. Further discussion of >> the proposal's details can happen on the tracker: >> >> http://bugs.python.org/issue0401 >> >> ChrisA > > > What's the actual issue number, I get "There is no issue with id 401" ? > > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From breamoreboy at yahoo.co.uk Wed Apr 1 23:31:44 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 01 Apr 2015 22:31:44 +0100 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On 01/04/2015 22:14, Serhiy Storchaka wrote: > On 02.04.15 00:05, Mark Lawrence wrote: >> On 01/04/2015 18:51, Chris Angelico wrote: >>> On Thu, Apr 2, 2015 at 4:32 AM, Ryan Gonzalez >>> wrote: >>>> Attached is a new module loader I've been working on that's an order of >>>> magnitude times faster than the current one. What would it take for >>>> this to >>>> be merged? >>> >>> LGTM. Before you can get this merged, you'll need to sign a >>> contributor agreement, if you haven't already. Further discussion of >>> the proposal's details can happen on the tracker: >>> >>> http://bugs.python.org/issue0401 >>> >>> ChrisA >> >> What's the actual issue number, I get "There is no issue with id 401" ? > > Cris missed. The correct link is http://bugs.python.org/issue0104. "There is no issue with id 104" > On 01/04/2015 22:14, Devin Jeanpierre wrote:> I think Chris meant to link to http://bugs.python.org/issue0404 , > sorry for any confusion. > > -- Devin > "There is no issue with id 404" Is this an April Fools? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From phd at phdru.name Wed Apr 1 23:35:03 2015 From: phd at phdru.name (Oleg Broytman) Date: Wed, 1 Apr 2015 23:35:03 +0200 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: <20150401213503.GA3111@phdru.name> On Wed, Apr 01, 2015 at 10:31:44PM +0100, Mark Lawrence wrote: > "There is no issue with id 404" > > Is this an April Fools? The day for the Error 404 is April 4. ;-) Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From jeanpierreda at gmail.com Wed Apr 1 23:43:50 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 1 Apr 2015 14:43:50 -0700 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On Wed, Apr 1, 2015 at 2:31 PM, Mark Lawrence wrote: > Is this an April Fools? Yes. The date is 04/01 or 01/04 depending on locale, and you are getting an HTTP 404. -- Devin From rymg19 at gmail.com Wed Apr 1 23:45:57 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 1 Apr 2015 16:45:57 -0500 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: You didn't open the archive, did you? ;) On Wed, Apr 1, 2015 at 4:31 PM, Mark Lawrence wrote: > On 01/04/2015 22:14, Serhiy Storchaka wrote: > >> On 02.04.15 00:05, Mark Lawrence wrote: >> >>> On 01/04/2015 18:51, Chris Angelico wrote: >>> >>>> On Thu, Apr 2, 2015 at 4:32 AM, Ryan Gonzalez >>>> wrote: >>>> >>>>> Attached is a new module loader I've been working on that's an order of >>>>> magnitude times faster than the current one. What would it take for >>>>> this to >>>>> be merged? >>>>> >>>> >>>> LGTM. Before you can get this merged, you'll need to sign a >>>> contributor agreement, if you haven't already. Further discussion of >>>> the proposal's details can happen on the tracker: >>>> >>>> http://bugs.python.org/issue0401 >>>> >>>> ChrisA >>>> >>> >>> What's the actual issue number, I get "There is no issue with id 401" ? >>> >> >> Cris missed. The correct link is http://bugs.python.org/issue0104. >> > > "There is no issue with id 104" > > On 01/04/2015 22:14, Devin Jeanpierre wrote:> I think Chris meant to link >> to http://bugs.python.org/issue0404 , >> sorry for any confusion. >> >> -- Devin >> >> > "There is no issue with id 404" > > Is this an April Fools? > > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 2 00:00:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 22:00:21 +0000 (UTC) Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: Message-ID: <1050504999.3255173.1427925621646.JavaMail.yahoo@mail.yahoo.com> On Wednesday, April 1, 2015 12:40 PM, Ron Adam wrote: > > On 04/01/2015 10:25 AM, Andrew Barnert wrote: >>> > xchg = [(124, 0x65), #LOAD_FAST to LOAD_NAME >>> > (125, 0x5a)] #STORE_FAST to STORE_NAME >> The first problem here is that any 124 or 125 in an operand to any >> opcode except 124 or 125 will be matched and converted (although > you'll usually >> probably get an IndexError trying to treat the next two arbitrary bytes as >> an index...). > > Yes, it will only work for very simple cases. It was just enough to get the > initial examples working. > >> To solve this, you need to iterate opcode by opcode, not byte by byte. >> The dis module gives you the information to tell how many bytes to skip for >> each opcode's operands. (It also maps between opcode numbers and names, > so >> you don't have to use magic numbers with comments.) Using it will solve >> this problem (and maybe others I didn't spot) and also make your code a > lot >> simpler. > > Unfortunately dis is written to give human output for python bytecode, not > to edit bytecode. But it can help. It needs a function to go back to a > code object after editing the instruction list. No; even without doing all the work for you, dis still provides more than enough information to be useful. For example: xchg = {'LOAD_FAST': 'LOAD_NAME', 'STORE_FAST': 'STORE_NAME'} bcode = bytearray(co_code) for instr in dis.Bytecode(co_code): try: newop = xchg[instr.opname] except KeyError: pass else: index = instr.arg char = varnames[index] names = names + (char,) index = names.index(char) b[instr.offset] = dis.opmap[newop] b[instr.offset+1:instr.offset+3] = struct.pack('>H', index) That does everything your big loop did, except that it doesn't generate crashing bytecode if you have 257 names or if any of your arguments are 124, 125, or [31744, 32256). (It still doesn't handle > 65536 names via EXTENDED_ARG, but last time I tested, albeit in 2.earlyish, neither did the interpreter itself, so that should be fine... If not, it's not too hard to add that too.) >> Why are you applying these to a dict? I thought the whole point was to >> be able to run it inside a scope and affect that scope's variables? If >> you just leave out the ns, won't that be closer to what you want? (And >> it also means you don't need the get_sig thing in the first place, and >> I'm not sure what that adds. Using a function signature plus a call >> expression as a fancy way of writing a dict display seems like just >> obfuscation. Maybe with default values, *args, binding partials and then >> calling get_sig on them, etc. is interesting for something, but I'm not >> sure what?) > > This was just a step in that direction. It obviously needs more work. > > There are a number of interesting aspects and directions this can go. > > * Ability to decompose functions into separate > signature and body parts. > > Not useful (or hard) in itself, but being able to reuse those parts may be > good. Again, why? If your goal is to be able to declare a body to be used inline in another scope, what will you ever need these signature objects for? > * Use those parts together. > > This is a basic test that should just work. Again it's not really that > useful in itself, but it's a nice to have equivalency and helps to > demonstrate how it works. > > > * Use one body with different signatures. > > For example you might have signatures with different default values, or > signatures that interface to different data formats. Rather than having a > function that converts different data formats to fit a single signature > format, we can just use different signatures to interface with the data > more directly. This is one of the things macro's do in other languages. All your signature objects can do is return a dict that you can eval a code block in, instead of evaling it in the current frame. If your goal is to eval it in the current frame, what good does that dict do you? > * Use code objects as blocks to implement continuation like behaviours. > > This is done breaking an algorithm into composable parts, then applying > them to data. It's not quite the same as continuations, or generators, but > has some of the same benefits. If the blocks avoid parsing signatures and > creating/destroying frames, it can be a fast way to translate data. Of > course, it's very limited as you need to have strict naming conventions to > do this. So it would be limited to within a scope that follows those > conventions. (Doable now with compile and exec, but it's awkward in my > opinion.) Sure, but again, the (transformed) code object alone already does exactly that. If the signature object added something (like being able to avoid the strict naming conventions, maybe?), it would be helpful, but it doesn't; you can do exactly the same thing with just the code object that you can do with both objects. > * Use a body as a block in another function. > > Yes, this requires getting the live namespace from the frame it's used in. That's trivial. If you just call eval with the default arguments, it gets the live namespace from the frame it's used in. If you want to wrap up eval in a function that does the exact same thing eval does, then you need to manually go up one frame. I'm not sure why you want to do that, but it's easy. > f = sys_getframe(-1) may work, but it's not that easy to do in this > case. def run_code_obj(code): loc = sys._getframe(-1).f_locals return eval(code, loc) How is that not easy? > When exec is given a code object, it call's PyEval_EvalCodeEx in > ceval.c directly with local and global dictionaries. (That should answer > some of your comments as to why I uses the dictionary.) Yes, and running the block of code directly with the local and global dictionaries is exactly what you want it to do, so why are you telling it not to do that? For example: def macro(): x += 1 code = fix_code(macro.__code__) def f(code_obj): x = 1 loc = locals() eval(code_obj) return loc['x'] (Or, if you prefer, use "run_code_obj" instead of "eval".) The problem here is that if you just "return x" at the end instead of "return loc['x']", you will likely see 1 instead of 2. It's the same problem you get if you "exec('x += 1')", exactly as described in the docs. That happens because f was compiled to look up x by index in the LOAD_FAST locals array, instead of by name in the locals dict, but your modified code objects mutate only the dict, not the array. That's the big problem you need to solve. Adding more layers of indirection doesn't get you any closer to fixing it. > It may be possible to call directly to PyEval_EvalFrameEx (like generators > do), with a frame object. No. You've already got access to exactly the same information that you'd have that way. The problem is that you converted all the STORE_FAST instructions to STORE_NAME, and that means you're ignoring the array of fast locals and only using the dict, which means that the calling code won't see your changes. One way you could solve this is by applying the same code conversion to any function that wants to _use_ a code block that you apply to one that wants to be _used as_ a code block (except the former would return wraps(types.FunctionType(code, f.__globals__, ...), ...) instead of just returning code). It's ugly, and it's a burden on the user, and it makes everything slower (and it may break functions that use normal closures plus your code block things), but it's the only thing that could possibly work. If you want to use STORE_NAME, the caller has to use LOAD_NAME. > Some or all of these may/will require the code object to be in a form that > is more easily relocatable, but as you noted, its not easy to do. If you want to be able to run these code blocks in unmodified functions (without radically changing the interpreter), then yes, you need to affect the caller's LOAD_FAST variables, which means you need to do a STORE_FAST with the caller's index for the variable, and you don't have the caller's index until call time, which means you need relocation. It isn't really _necessary_ to make the code easily relocatable, it just makes the relocation (which is necessary) easier and more efficient. For example, at definition time, you can build a table like: {'x': (1, 8)} So at call time, all you have to do is: names = {name: index for index, name in enumerate(sys._getframe(-1).f_code.co_names)} b = bytearray(c.co_code) for name, offsets in relocs.items(): index = names[name] for offset in offsets: b[offset:offset+2] = struct.pack('>H', index) code = types.CodeType(blah, blah, bytes(b), blah) (Note that this will raise a KeyError if the called code block references a variable that doesn't exist in the calling scope; you may want to catch that and reraise it as a different exception. Also note that, as I explained before, you may want to map NAME/GLOBAL/CELL lookups to FAST lookups--almost the exact opposite of what you're doing--so that code like "def f(): return x+1" sees the calling function's local x, not the global or cell x at definition time, but that's tricky because you probably want "def f(): global x; return x+1" to see the global x...) _Now_ you face the problem that you need to run this on the actual calling frame, rather than what exec does (effectively, run it on a temporary frame with the same locals and globals dicts). And I think that will require extending the interpreter. But all the stuff you wrote above isn't a step in the direction of doing that, it's a step _away_ from that. Once you have that new functionality, you will not want a code object that's converted all FAST variables to NAME variables, or a signature object that gives you a different set of locals to use than the ones you want, or anything like that; you will want a code object that leaves FAST variables as FAST variables but renumbers them, and uses the frame's variables rather than a different namespace. > There are a lot of 'ifs' here, but I think it may be worth exploring. > > I'm going to try and make the bytecode fixer function work better (using > dis or parts of it.) And then put it up on github where this idea can be > developed further. > > (The other utilities I've found so far for editing bytecode aren't > ported > to python3 yet.) As I said in my previous message, there are at least three incomplete ports of byteplay to 3.x. I think https://github.com/serprex/byteplay works on 3.2, but not 3.3 or 3.4. https://github.com/abarnert/byteplay works on 3.4, but mishandles certain constructions where 2.7+/3.3+ optimizes try/with statements (try running it on the stdlib, and you'll see exceptions on three modules) that I'm pretty sure won't affect your examples. At any rate, while fixing and using byteplay (or replacing it with something new that requires 3.4+ dis, or 2.7/3.3 with the dis 3.4 backport, and avoids all the hacky mucking around trying to guess at stack effects) might make your code nicer, I don't think it's necessary; what you need is a small subset of what it can do (e.g., you're not inserting new instructions and renumbering all the jump offsets, or adding wrapping statements in try blocks, etc.), so you could just cannibalize it to borrow the parts you need and ignore the rest. > I don't think editing the bytecode is the ideal solution in the long run, > but it will identify the parts that need addressing and then other > solutions for those could be looked into such as doing the needed > alterations in the AST rather than the bytecode. If you want to be able to convert functions to code blocks at runtime (which is inherent in using a decorator), the bytecode is all you have. If you want to capture the AST, you need to do at import/compile time. If you're going to do that, MacroPy already does an amazing job of that, so why reinvent the wheel? (If there's some specific problem with MacroPy that you don't think can be solved without a major rearchitecture, I'll bet Haoyi Li would like to know about it...) And, more importantly, why put all this work into something completely different, which has a completely different set of problems to solve, if you're just going to throw it out later? For example, all the problems with renumbering variables indices or converting between different kinds of variables that you're solving here won't help you identify anything relevant to an AST-based solution, where variables are still just Name(id='x'). From abarnert at yahoo.com Thu Apr 2 00:13:21 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 15:13:21 -0700 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: I tried this on Windows, and it has a serious Hanlon's Razor bug. Sent from my iPhone > On Apr 1, 2015, at 10:32, Ryan Gonzalez wrote: > > Attached is a new module loader I've been working on that's an order of magnitude times faster than the current one. What would it take for this to be merged? > > BTW... > > (rot13) Ncevy Sbbyf! > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. > http://kirbyfan64.github.io/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 2 00:17:55 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 1 Apr 2015 15:17:55 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <1050504999.3255173.1427925621646.JavaMail.yahoo@mail.yahoo.com> References: <1050504999.3255173.1427925621646.JavaMail.yahoo@mail.yahoo.com> Message-ID: On Apr 1, 2015 3:02 PM, "Andrew Barnert" wrote: > > On Wednesday, April 1, 2015 12:40 PM, Ron Adam wrote: > > When exec is given a code object, it call's PyEval_EvalCodeEx in > > ceval.c directly with local and global dictionaries. (That should answer > > some of your comments as to why I uses the dictionary.) > > Yes, and running the block of code directly with the local and global dictionaries is exactly what you want it to do, so why are you telling it not to do that? > > For example: > > > def macro(): > x += 1 > code = fix_code(macro.__code__) > > def f(code_obj): > x = 1 > loc = locals() > eval(code_obj) > return loc['x'] > > (Or, if you prefer, use "run_code_obj" instead of "eval".) > > The problem here is that if you just "return x" at the end instead of "return loc['x']", you will likely see 1 instead of 2. It's the same problem you get if you "exec('x += 1')", exactly as described in the docs. > > That happens because f was compiled to look up x by index in the LOAD_FAST locals array, instead of by name in the locals dict, but your modified code objects mutate only the dict, not the array. That's the big problem you need to solve. Adding more layers of indirection doesn't get you any closer to fixing it. You can propagate changes to the dict back to the array by calling the c api function PyFrame_LocalsToDict. It's pretty easy to do via ctypes, see e.g. http://pydev.blogspot.com/2014/02/changing-locals-of-frame-frameflocals.html?m=1 I guess you could append some byte code to do this to your modified function bodies. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Apr 2 00:35:41 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 1 Apr 2015 17:35:41 -0500 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On MSVC? If you want, you can remove the #ifdef's for MSVC. They're kind of useless. I just put them because it's funny. :) On Wed, Apr 1, 2015 at 5:13 PM, Andrew Barnert wrote: > I tried this on Windows, and it has a serious Hanlon's Razor bug. > > Sent from my iPhone > > On Apr 1, 2015, at 10:32, Ryan Gonzalez wrote: > > Attached is a new module loader I've been working on that's an order of > magnitude times faster than the current one. What would it take for this to > be merged? > > BTW... > > (rot13) Ncevy Sbbyf! > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 2 01:06:14 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 16:06:14 -0700 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On Apr 1, 2015, at 15:35, Ryan Gonzalez wrote: > > On MSVC? > > If you want, you can remove the #ifdef's for MSVC. They're kind of useless. I just put them because it's funny. :) Google Hanlon's Razor, and I think you'll understand how to fix the bug. > On Wed, Apr 1, 2015 at 5:13 PM, Andrew Barnert wrote: >> I tried this on Windows, and it has a serious Hanlon's Razor bug. >> >> Sent from my iPhone >> >>> On Apr 1, 2015, at 10:32, Ryan Gonzalez wrote: >>> >>> Attached is a new module loader I've been working on that's an order of magnitude times faster than the current one. What would it take for this to be merged? >>> >>> BTW... >>> >>> (rot13) Ncevy Sbbyf! >>> >>> -- >>> Ryan >>> [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. >>> http://kirbyfan64.github.io/ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. > http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 2 01:05:27 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 16:05:27 -0700 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: References: <1050504999.3255173.1427925621646.JavaMail.yahoo@mail.yahoo.com> Message-ID: <28357B39-8BED-4588-B32A-0E8F15DAE1E7@yahoo.com> On Apr 1, 2015, at 15:17, Nathaniel Smith wrote: > > On Apr 1, 2015 3:02 PM, "Andrew Barnert" wrote: > > > > On Wednesday, April 1, 2015 12:40 PM, Ron Adam wrote: > > > When exec is given a code object, it call's PyEval_EvalCodeEx in > > > ceval.c directly with local and global dictionaries. (That should answer > > > some of your comments as to why I uses the dictionary.) > > > > Yes, and running the block of code directly with the local and global dictionaries is exactly what you want it to do, so why are you telling it not to do that? > > > > For example: > > > > > > def macro(): > > x += 1 > > code = fix_code(macro.__code__) > > > > def f(code_obj): > > x = 1 > > loc = locals() > > eval(code_obj) > > return loc['x'] > > > > (Or, if you prefer, use "run_code_obj" instead of "eval".) > > > > The problem here is that if you just "return x" at the end instead of "return loc['x']", you will likely see 1 instead of 2. It's the same problem you get if you "exec('x += 1')", exactly as described in the docs. > > > > That happens because f was compiled to look up x by index in the LOAD_FAST locals array, instead of by name in the locals dict, but your modified code objects mutate only the dict, not the array. That's the big problem you need to solve. Adding more layers of indirection doesn't get you any closer to fixing it. > > You can propagate changes to the dict back to the array by calling the c api function PyFrame_LocalsToDict. It's pretty easy to do via ctypes, see e.g. > You mean PyFrame_LocalsToFast, not the other way around, right? That's a good idea. There might be problems executing two code blocks (or a code block and a normal eval/exec statement) in the same function, but for a prototype that's fine... > http://pydev.blogspot.com/2014/02/changing-locals-of-frame-frameflocals.html?m=1 > > I guess you could append some byte code to do this to your modified function bodies. > That would be painful without byteplay--you have to insert the new instructions before every return and raise bytecode, which means renumbering jumps, etc. But do you really need to? Can you do it in the wrapper? def call_code(code): frame = sys._getframe(1) PyFrame_LocalsToDict(py_object(frame)) try: return eval(code, frame.f_locals(), frame.f_globals()) finally: PyFrame_LocalsToFast(py_object(frame)) (Doing the matched pair like this might avoid the problem with multiple code blocks in one function. I'm not sure, but... worth a try, right?) I think there will still be problem with cell vars (that is, updating a local in the caller which is used in a closure by a local function in the caller). And there's definitely still the problem of magically guessing which variables are meant to be local, closure, or global. But again, for a prototype, that all may be fine. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Apr 2 01:30:49 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 1 Apr 2015 18:30:49 -0500 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: Since April Fools is almost over, I might as well post the GitHub link: https://github.com/kirbyfan64/pycob Have fun! On Wed, Apr 1, 2015 at 12:32 PM, Ryan Gonzalez wrote: > Attached is a new module loader I've been working on that's an order of > magnitude times faster than the current one. What would it take for this to > be merged? > > BTW... > > (rot13) Ncevy Sbbyf! > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Thu Apr 2 01:27:58 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 1 Apr 2015 18:27:58 -0500 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: I did. Most of it went over my head... On Wed, Apr 1, 2015 at 6:06 PM, Andrew Barnert wrote: > On Apr 1, 2015, at 15:35, Ryan Gonzalez wrote: > > On MSVC? > > If you want, you can remove the #ifdef's for MSVC. They're kind of > useless. I just put them because it's funny. :) > > > Google Hanlon's Razor, and I think you'll understand how to fix the bug. > > On Wed, Apr 1, 2015 at 5:13 PM, Andrew Barnert wrote: > >> I tried this on Windows, and it has a serious Hanlon's Razor bug. >> >> Sent from my iPhone >> >> On Apr 1, 2015, at 10:32, Ryan Gonzalez wrote: >> >> Attached is a new module loader I've been working on that's an order of >> magnitude times faster than the current one. What would it take for this to >> be merged? >> >> BTW... >> >> (rot13) Ncevy Sbbyf! >> >> -- >> Ryan >> [ERROR]: Your autotools build scripts are 200 lines longer than your >> program. Something?s wrong. >> http://kirbyfan64.github.io/ >> >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Apr 2 01:48:35 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 16:48:35 -0700 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On Apr 1, 2015, at 16:27, Ryan Gonzalez wrote: > > I did. Most of it went over my head... Ha, you're just pretending to be stupid as part of your evil plan. Err... Wait a second... >> On Wed, Apr 1, 2015 at 6:06 PM, Andrew Barnert wrote: >>> On Apr 1, 2015, at 15:35, Ryan Gonzalez wrote: >>> >>> On MSVC? >>> >>> If you want, you can remove the #ifdef's for MSVC. They're kind of useless. I just put them because it's funny. :) >> >> Google Hanlon's Razor, and I think you'll understand how to fix the bug. >> >>>> On Wed, Apr 1, 2015 at 5:13 PM, Andrew Barnert wrote: >>>> I tried this on Windows, and it has a serious Hanlon's Razor bug. >>>> >>>> Sent from my iPhone >>>> >>>>> On Apr 1, 2015, at 10:32, Ryan Gonzalez wrote: >>>>> >>>>> Attached is a new module loader I've been working on that's an order of magnitude times faster than the current one. What would it take for this to be merged? >>>>> >>>>> BTW... >>>>> >>>>> (rot13) Ncevy Sbbyf! >>>>> >>>>> -- >>>>> Ryan >>>>> [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. >>>>> http://kirbyfan64.github.io/ >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >>> >>> -- >>> Ryan >>> [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. >>> http://kirbyfan64.github.io/ > > > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. > http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Thu Apr 2 02:11:57 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Thu, 02 Apr 2015 01:11:57 +0100 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On 01/04/2015 22:45, Ryan Gonzalez wrote: > You didn't open the archive, did you? ;) > No, mainly because autistic people like myself don't like having our time wasted with crap like this. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From steve at pearwood.info Thu Apr 2 03:13:59 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 2 Apr 2015 12:13:59 +1100 Subject: [Python-ideas] History on proposals for Macros? In-Reply-To: <1050504999.3255173.1427925621646.JavaMail.yahoo@mail.yahoo.com> References: <1050504999.3255173.1427925621646.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20150402011359.GW25453@ando.pearwood.info> On Wed, Apr 01, 2015 at 10:00:21PM +0000, Andrew Barnert wrote: [mass snippage deployed] While I do have some interest in this subject, I think at the point you are doing detailed code reviews of experimental software, it's probably no longer on-topic for this mailing list and possibly should be taken off-list until you have something concrete to report. Also, if you want to explore this further: (1) Hacking the byte-code is not portable. It won't work in non-CPython implementations, and bytecode is not a stable part of the CPython API either. Hacking the AST may be better. (2) If you must hack the bytecode, there is at least one library for bytecode manipulations out in there, possibly on PyPI. Google for "python byte-code hacking" for more. -- Steve From rymg19 at gmail.com Thu Apr 2 04:30:36 2015 From: rymg19 at gmail.com (Ryan) Date: Wed, 01 Apr 2015 21:30:36 -0500 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: Andrew Barnert wrote: >On Apr 1, 2015, at 16:27, Ryan Gonzalez wrote: >> >> I did. Most of it went over my head... > >Ha, you're just pretending to be stupid as part of your evil plan. > >Err... Wait a second... ???? No, really. I have no clue what your point is. > >>> On Wed, Apr 1, 2015 at 6:06 PM, Andrew Barnert >wrote: >>>> On Apr 1, 2015, at 15:35, Ryan Gonzalez wrote: >>>> >>>> On MSVC? >>>> >>>> If you want, you can remove the #ifdef's for MSVC. They're kind of >useless. I just put them because it's funny. :) >>> >>> Google Hanlon's Razor, and I think you'll understand how to fix the >bug. >>> >>>>> On Wed, Apr 1, 2015 at 5:13 PM, Andrew Barnert > wrote: >>>>> I tried this on Windows, and it has a serious Hanlon's Razor bug. >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Apr 1, 2015, at 10:32, Ryan Gonzalez wrote: >>>>>> >>>>>> Attached is a new module loader I've been working on that's an >order of magnitude times faster than the current one. What would it >take for this to be merged? >>>>>> >>>>>> BTW... >>>>>> >>>>>> (rot13) Ncevy Sbbyf! >>>>>> >>>>>> -- >>>>>> Ryan >>>>>> [ERROR]: Your autotools build scripts are 200 lines longer than >your program. Something?s wrong. >>>>>> http://kirbyfan64.github.io/ >>>>>> >>>>>> _______________________________________________ >>>>>> Python-ideas mailing list >>>>>> Python-ideas at python.org >>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>>> >>>> >>>> -- >>>> Ryan >>>> [ERROR]: Your autotools build scripts are 200 lines longer than >your program. Something?s wrong. >>>> http://kirbyfan64.github.io/ >> >> >> >> -- >> Ryan >> [ERROR]: Your autotools build scripts are 200 lines longer than your >program. Something?s wrong. >> http://kirbyfan64.github.io/ -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Check out my website: http://kirbyfan64.github.io/ From tjreedy at udel.edu Thu Apr 2 05:04:50 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Apr 2015 23:04:50 -0400 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On 4/1/2015 5:45 PM, Ryan Gonzalez wrote: > You didn't open the archive, did you? ;) .tar files are not Windows friendly so I skipped it. -- Terry Jan Reedy From rosuav at gmail.com Thu Apr 2 05:33:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 2 Apr 2015 14:33:23 +1100 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On Thu, Apr 2, 2015 at 2:04 PM, Terry Reedy wrote: > On 4/1/2015 5:45 PM, Ryan Gonzalez wrote: >> >> You didn't open the archive, did you? ;) > > > .tar files are not Windows friendly > so I skipped it. 7-zip is Windows-friendly and tar-friendly. You could ask for a social introduction. "Mr Tar, may I present Ms Windows?" ChrisA From abarnert at yahoo.com Thu Apr 2 06:05:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 1 Apr 2015 21:05:47 -0700 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On Apr 1, 2015, at 19:30, Ryan wrote: > > > Andrew Barnert wrote: >>> On Apr 1, 2015, at 16:27, Ryan Gonzalez wrote: >>> >>> I did. Most of it went over my head... >> >> Ha, you're just pretending to be stupid as part of your evil plan. >> >> Err... Wait a second... > > ???? > > No, really. I have no clue what your point is. Hanlon's Razor says "Never attribute to evil that which is plausibly explained by stupidity." So, the fix is simple: 's/is evil/is stupid/g'. (Well, it would be simple if Windows came with sed. Or patch. Or... But if you have MSVC, you can buy a $30 tool that will simulate hitting ctrl-shift-F, clicking in the text boxes, typing the text, and clicking the three buttons, which is just as good, right?) >>>> On Wed, Apr 1, 2015 at 6:06 PM, Andrew Barnert >> wrote: >>>>> On Apr 1, 2015, at 15:35, Ryan Gonzalez wrote: >>>>> >>>>> On MSVC? >>>>> >>>>> If you want, you can remove the #ifdef's for MSVC. They're kind of >> useless. I just put them because it's funny. :) >>>> >>>> Google Hanlon's Razor, and I think you'll understand how to fix the >> bug. >>>> >>>>>> On Wed, Apr 1, 2015 at 5:13 PM, Andrew Barnert >> wrote: >>>>>> I tried this on Windows, and it has a serious Hanlon's Razor bug. >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On Apr 1, 2015, at 10:32, Ryan Gonzalez wrote: >>>>>>> >>>>>>> Attached is a new module loader I've been working on that's an >> order of magnitude times faster than the current one. What would it >> take for this to be merged? >>>>>>> >>>>>>> BTW... >>>>>>> >>>>>>> (rot13) Ncevy Sbbyf! >>>>>>> >>>>>>> -- >>>>>>> Ryan >>>>>>> [ERROR]: Your autotools build scripts are 200 lines longer than >> your program. Something?s wrong. >>>>>>> http://kirbyfan64.github.io/ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Python-ideas mailing list >>>>>>> Python-ideas at python.org >>>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>>> >>>>> >>>>> -- >>>>> Ryan >>>>> [ERROR]: Your autotools build scripts are 200 lines longer than >> your program. Something?s wrong. >>>>> http://kirbyfan64.github.io/ >>> >>> >>> >>> -- >>> Ryan >>> [ERROR]: Your autotools build scripts are 200 lines longer than your >> program. Something?s wrong. >>> http://kirbyfan64.github.io/ > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > Check out my website: http://kirbyfan64.github.io/ From songofacandy at gmail.com Thu Apr 2 12:54:07 2015 From: songofacandy at gmail.com (Naoki INADA) Date: Thu, 2 Apr 2015 03:54:07 -0700 (PDT) Subject: [Python-ideas] Adding rule about trailing comma. In-Reply-To: References: Message-ID: One good parts of Go is gofmt, the standard code formatter. It makes learning Go more easy and fun. New Gophers can learn Go without learning coding style which gofmt handles. Python doesn't have standard code formatter like gofmt, but there are some third party checkers and formatters including pep8.py, autopep8 and google/yapf. There are also formmatters integrated in IDEs. My concern is defining good edge cases for code formatter developers. So my real question was: How many people prefer (or dislike) this rule is in code formatter? If many people prefer this, could we have standard "Style guide for code formatter developers" for rules too verbose to include PEP 8? On Tuesday, March 31, 2015 at 5:34:40 PM UTC+9, Chris Angelico wrote: > > On Tue, Mar 31, 2015 at 7:28 PM, INADA Naoki > wrote: > > Benefit of this style is: > > > > 1) More consistent > > 2) No garbage diff just for just adding comma > > So what you have there is a good reason for adopting this style. Is > there strong reason for mentioning it in PEP 8? Remember, every > recommendation in a style guide has a cost - too many rules and it > just becomes onerous, not to mention the increased likelihood of the > rule giving the wrong advice in some corner case (which leads > intelligent people to disregard the rule, and unintelligent checkers > to spew warnings which then have to be ignored - this is one of the > fundamental difficulties of scripting PEP 8 or any other style guide); > ideally, trust people to be intelligent, and don't enforce more than > you have to. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Thu Apr 2 13:51:28 2015 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Thu, 2 Apr 2015 22:51:28 +1100 Subject: [Python-ideas] A new module loader In-Reply-To: References: Message-ID: On 2 April 2015 at 14:33, Chris Angelico wrote: > On Thu, Apr 2, 2015 at 2:04 PM, Terry Reedy wrote: > > On 4/1/2015 5:45 PM, Ryan Gonzalez wrote: > >> > >> You didn't open the archive, did you? ;) > > > > > > .tar files are not Windows friendly > > so I skipped it. > > 7-zip is Windows-friendly and tar-friendly. You could ask for a social > introduction. > Unfortunately, 7-Zip does not support the PAX extended header (e.g. for long filenames). I'm not sure about the oldgnu format off the top of my head. Fortunately, the MinGW tools do. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Apr 2 21:01:08 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 2 Apr 2015 21:01:08 +0200 Subject: [Python-ideas] Clear all caches In-Reply-To: References: Message-ID: Hi, 2015-04-01 14:28 GMT+02:00 Serhiy Storchaka : > There are a lot of implicit caches in the stdlib. Some of them can grow > unlimitedly. It would be good to have an official way to clear caches. > > I proposed to add two functions (in some existing lightweight module or add > a new module): I proposed a similar idea 2 years ago: https://mail.python.org/pipermail/python-dev/2013-October/129218.html Victor From techtonik at gmail.com Fri Apr 3 10:33:09 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 3 Apr 2015 11:33:09 +0300 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Fri, Mar 27, 2015 at 7:55 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Fri, Mar 27, 2015 at 12:13 PM, anatoly techtonik > wrote: > > > > Here is something that can be used as an example that it is not about > PEP8 https://code.google.com/p/rainforce/wiki/WartsOfPython#measure_time > And it takes a lot of energy to collect something like that for the > reference. > > > Well, in that document, I see the total of four "warts" related to the > datetime module. If four warts bring a module to the top of the "worst > designed stdlib modules" list, it can only mean that stdlib is almost > perfect! > This is just a personal list of one person for the stuff that is the most annoying to be worthy of spending 4 hours on research and documentation. I've spent at least one working day tracing and recording this down, needless to say that I had to invent the whole concept of wart for the stuff that is neither a bug, nor a feature. The quantity argument that you using doesn't not apply to usability issues. Take any of your gadgets for example. Remember non-USB cables for your phone? How you'd be running around and asking people if anybody has Nokia or Sony or whatever-brand-is-so-smart cable around. This is the wart, and no matter how do you like the phone and your brand and justify Apple use their own cables, for the rest of the world with normal phones this looks weird. > For each "wart" the author has a "What can be done?" section, but no > suggested solution involves a major redesign of the datetime module. > Author is me, so you can ask directly. Why I didn't propose to redesign? Because people will assume that somebody will need to write PEP and will force me to write one. I don't believe in "redesign by specification" like current PEP process assumes and people accuse me of being lazy and trolling them, because I don't want to write the PEPs. Damn, I believe in iterative development and evolution, and I failed to persuade coredevs that practices digged up by people under the "agile" label is not some sort of corporate bullshit. So it is not my problem now. I did all I am capable of. The author complains about non-obviousness of strftime and indeed, for > users without C background, neither name nor semantics is familiar. But > the proposed solution > > >>> time.format('{{hours}}:{{minutes}}:{{seconds}}', 1090) > '00:18:10' > > > Does not look like a big win over the already available solution: > > >>> '{t.hour}:{t.minute}:{t.second}'.format(t=datetime.now()) > '12:37:38' > You're evaluating the "big win" using your own biased criteria, not the one that are used by OP. I do not see how this solution makes statement "no obvious way to format time in seconds" false. > Overall, while I agree that there are a few warts in the datetime module, > I have not seen any serious enough to warrant a major redesign or even any > non backward compatible changes. Maintaining backward compatibility in the > datetime module is indeed non-trivial, but this is the way I think it > should be developed. > You're hijacking the issue into BC black hole leaving no chance to provide a better alternative. Let's state that it is obvious that the *behaviour of modules that need a redesign will not change* - the best thing that could happen is that they will get replacements without chronic disease encoded in their DNA. And to engineer that DNA, a proper "scientific method" should be used. "Writing a PEP" is not a method, and "it works for me" is not an argument. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Fri Apr 3 10:44:58 2015 From: techtonik at gmail.com (anatoly techtonik) Date: Fri, 3 Apr 2015 11:44:58 +0300 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <6F7DC05A-2563-4DD1-B20A-F8F1C9D37630@yahoo.com> References: <6F7DC05A-2563-4DD1-B20A-F8F1C9D37630@yahoo.com> Message-ID: On Sat, Mar 28, 2015 at 3:21 AM, Andrew Barnert wrote: > On Mar 27, 2015, at 08:21, Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > > > On Fri, Mar 27, 2015 at 10:54 AM, anatoly techtonik > wrote: > >> Where have you been when PEP 3108 was discussed? I have not seen any >>> other list of Python modules that needed a redesign, so I cannot tell >>> what's on your top ten list. >>> >> >> http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html >> > > Interesting. I did not see this back in the day. The top entry (urllib > and friends) makes sense and there were heavily redesigned in Python 3. I > am surprised that distutils redesign got less support than logging and > datetime. > > > I'm not sure how much weight to put on an informal poll of 176 people > self-selected as readers of a particular blog in the first place, but... > > Back in those days, I think most people had no idea what they wanted from > a redesign of distutils, and until at least one of the competing projects > to extend/fix/replace it was ready for prime time (at least design-wise) > the idea of changing the stdlib to follow one of them was a bit scary. > You're absolutely right that 176 people is hardly a honest poll. It is just an example that this thing is useful, so in ideal world PSF contact somebody who did that and support him with contacts, money, space, and other resources necessary to bring this to the level of representative and authoritative research. > I suspect that this may have to do with logging and datetime APIs not > being PEP8 compliant. Popular votes tend to exaggerate the importance of > trivial things such as the spelling of class and method names and brush off > more subtle, but important design flaws. > > > Interesting point. And I think you're right--but I think you can take it > farther. > > Even beyond the PEP 8 renaming, what kinds of things did people really > want from those modules? People also wanted the 90%-compatible > formatting/parsing functions to be 100% Java-compatible for logging and > 100% my-plafform's-C-lib-compatible for datetime. And they wanted to be > able to plug pytz into datetime without making it so easy to write what > looks like correct timezone-aware code but actually isn't. And they wanted > easier conversion between datetime's types and those in time and elsewhere. > > Those changes aren't as trivial as PEP 8 renaming, but they're still > simple to express, concrete, and unambiguous (while still potentially > requiring a backward-incompatible change). Who wouldn't vote for that? > I totally agree with all above, except that "Those changes aren't as trivial as PEP 8 renaming, but they're still simple to express" - my opinion is that they are not simple to express. If something is missing, it is absolutely not easy to spot and it requires relatively a lot of time to reach a level of confidence that "no, that's impossible". Thanks to StackOverflow, it is less of a problem today. And still, while it is possible to *express* the changes or the problem somewhere in mailing list, it is absolutely impossible to make an **action item** or at least a **concept to consider for module redesign** out of that. Because you know that happens to those issues and bugs - they are closed, because "it works for me" or "it will break backward compatibility". -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 4 06:18:24 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Apr 2015 15:18:24 +1100 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Fri, Apr 3, 2015 at 7:33 PM, anatoly techtonik wrote: > Author is me, so you can ask directly. Why I didn't propose to redesign? > Because people will assume that somebody will need to write PEP and will > force me to write one. I don't believe in "redesign by specification" like > current PEP process assumes and people accuse me of being lazy and trolling > them, because I don't want to write the PEPs. Damn, I believe in iterative > development and evolution, and I failed to persuade coredevs that practices > digged up by people under the "agile" label is not some sort of corporate > bullshit. So it is not my problem now. I did all I am capable of. Why, exactly, is it that you don't want to author a PEP? Is it because you don't have the time to devote to chairing the discussion and all? If so, you could quite possibly persuade someone else to. I'd be willing to take on the job; convince me that your core idea is worth pursuing (and make clear to me precisely what your core idea is), and I could do the grunt-work of writing. But you say that you "don't *believe in*" the process, which suggests a more philosophical objection. What's the issue, here? Why are you holding back from such a plan? *cue the troll music* There are many Pythons in the world. You can't just hack on CPython and expect everything to follow on from there. Someone has to explain to the Jython folks what they'll have to do to be compatible. Someone has to write something up so MicroPython can run the same code that CPython does. Someone, somewhere, has to be able to ensure that Brython users aren't caught out by your proposed change. PEPs provide that. (They also provide useful pointers for the "What's New" lists, eg PEP 441.) So, are you proposing a change to Python? Then propose it. ChrisA From breamoreboy at yahoo.co.uk Sat Apr 4 20:25:23 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 04 Apr 2015 19:25:23 +0100 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On 04/04/2015 05:18, Chris Angelico wrote: > On Fri, Apr 3, 2015 at 7:33 PM, anatoly techtonik wrote: >> Author is me, so you can ask directly. Why I didn't propose to redesign? >> Because people will assume that somebody will need to write PEP and will >> force me to write one. I don't believe in "redesign by specification" like >> current PEP process assumes and people accuse me of being lazy and trolling >> them, because I don't want to write the PEPs. Damn, I believe in iterative >> development and evolution, and I failed to persuade coredevs that practices >> digged up by people under the "agile" label is not some sort of corporate >> bullshit. So it is not my problem now. I did all I am capable of. > > Why, exactly, is it that you don't want to author a PEP? Is it because > you don't have the time to devote to chairing the discussion and all? > If so, you could quite possibly persuade someone else to. I'd be > willing to take on the job; convince me that your core idea is worth > pursuing (and make clear to me precisely what your core idea is), and > I could do the grunt-work of writing. But you say that you "don't > *believe in*" the process, which suggests a more philosophical > objection. What's the issue, here? Why are you holding back from such > a plan? *cue the troll music* > > There are many Pythons in the world. You can't just hack on CPython > and expect everything to follow on from there. Someone has to explain > to the Jython folks what they'll have to do to be compatible. Someone > has to write something up so MicroPython can run the same code that > CPython does. Someone, somewhere, has to be able to ensure that > Brython users aren't caught out by your proposed change. PEPs provide > that. (They also provide useful pointers for the "What's New" lists, > eg PEP 441.) > > So, are you proposing a change to Python? Then propose it. > > ChrisA > I don't understand why people bother with this gentleman. All talk, no action, but expects others to do his bidding. I would say "Please go take a running jump", but that would get me into trouble with the CoC aficionados, so I won't. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From alexander.belopolsky at gmail.com Sat Apr 4 22:16:47 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 4 Apr 2015 16:16:47 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: On Sat, Apr 4, 2015 at 2:25 PM, Mark Lawrence wrote: > > I don't understand why people bother with this gentleman. I did see constructive input from Anatoly on the bug tracker before. I believe other developers did too. Granted, in the last few years signal-to-noise ratio in his posts was rather low, but I usually ignore the "from:" header in the technical discussions. I am genuinely interested in the ways to improve date/time formatting in Python. There are certainly better ways than stftime. For example, ICU has a date/time format syntax that is much more readable: instead of "%Y-%m-%d %H:%M:%S", ICU format is "yyyy-MM-dd HH:mm:ss". I don't think it is hard to find a way to introduce ICU format in datetime.__format__ while preserving backward compatibility. For example, we may require that % is always "escaped" as '%' in ICU format and the presence of unescaped % can trigger strftime interpretation. [1] http://userguide.icu-project.org/formatparse/datetime -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Apr 5 01:28:18 2015 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 05 Apr 2015 00:28:18 +0100 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: <55207392.8060608@mrabarnett.plus.com> On 2015-04-04 21:16, Alexander Belopolsky wrote: > > On Sat, Apr 4, 2015 at 2:25 PM, Mark Lawrence > wrote: > > > > I don't understand why people bother with this gentleman. > > > I did see constructive input from Anatoly on the bug tracker before. I > believe other developers did too. Granted, in the last few years > signal-to-noise ratio in his posts was rather low, but I usually ignore > the "from:" header in the technical discussions. > > I am genuinely interested in the ways to improve date/time formatting in > Python. There are certainly better ways than stftime. For example, ICU > has a date/time format syntax that is much more readable: instead of > "%Y-%m-%d %H:%M:%S", ICU format is "yyyy-MM-dd HH:mm:ss". > > I don't think it is hard to find a way to introduce ICU format in > datetime.__format__ while preserving backward compatibility. For > example, we may require that % is always "escaped" as '%' in ICU format > and the presence of unescaped % can trigger strftime interpretation. > > > [1] http://userguide.icu-project.org/formatparse/datetime > Personally, I prefer some kind of escaping, possibly in the style of .format, e.g. "{year}-{month}-{day} {hour}:{minute}:{second}". (It'll probably need a little tinkering to shorten it! :-)) From alexander.belopolsky at gmail.com Sun Apr 5 02:45:03 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 4 Apr 2015 20:45:03 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <55207392.8060608@mrabarnett.plus.com> References: <55207392.8060608@mrabarnett.plus.com> Message-ID: On Sat, Apr 4, 2015 at 7:28 PM, MRAB wrote: > I am genuinely interested in the ways to improve date/time formatting in >> Python. There are certainly better ways than stftime. For example, ICU >> has a date/time format syntax that is much more readable: instead of >> "%Y-%m-%d %H:%M:%S", ICU format is "yyyy-MM-dd HH:mm:ss". >> >> I don't think it is hard to find a way to introduce ICU format in >> datetime.__format__ while preserving backward compatibility. For >> example, we may require that % is always "escaped" as '%' in ICU format >> and the presence of unescaped % can trigger strftime interpretation. >> >> >> [1] http://userguide.icu-project.org/formatparse/datetime >> >> Personally, I prefer some kind of escaping, possibly in the style of > .format, e.g. "{year}-{month}-{day} {hour}:{minute}:{second}". (It'll > probably need a little tinkering to shorten it! :-)) Can someone explain to me why something like this or Anatoly's double-curly-brace variant is an improvement over >>> from datetime import * >>> "{0.year}-{0.month}-{0.day} {0.hour}:{0.minute}:{0.second}".format(datetime.now()) '2015-4-4 20:43:23' -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Apr 4 09:22:47 2015 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 4 Apr 2015 00:22:47 -0700 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: Note also that a PEP does not need to be the first step. Write the code, ask people to try it out, if others like it, they may test and contribute, etc. While a PEP may be necessary to get something into the stdlib or core, it can be a document that captures the interactive, "agile" process -- it does not need to be design up-front. -CHB On Saturday, April 4, 2015, Chris Angelico wrote: > On Fri, Apr 3, 2015 at 7:33 PM, anatoly techtonik > wrote: > > Author is me, so you can ask directly. Why I didn't propose to redesign? > > Because people will assume that somebody will need to write PEP and will > > force me to write one. I don't believe in "redesign by specification" > like > > current PEP process assumes and people accuse me of being lazy and > trolling > > them, because I don't want to write the PEPs. Damn, I believe in > iterative > > development and evolution, and I failed to persuade coredevs that > practices > > digged up by people under the "agile" label is not some sort of corporate > > bullshit. So it is not my problem now. I did all I am capable of. > > Why, exactly, is it that you don't want to author a PEP? Is it because > you don't have the time to devote to chairing the discussion and all? > If so, you could quite possibly persuade someone else to. I'd be > willing to take on the job; convince me that your core idea is worth > pursuing (and make clear to me precisely what your core idea is), and > I could do the grunt-work of writing. But you say that you "don't > *believe in*" the process, which suggests a more philosophical > objection. What's the issue, here? Why are you holding back from such > a plan? *cue the troll music* > > There are many Pythons in the world. You can't just hack on CPython > and expect everything to follow on from there. Someone has to explain > to the Jython folks what they'll have to do to be compatible. Someone > has to write something up so MicroPython can run the same code that > CPython does. Someone, somewhere, has to be able to ensure that > Brython users aren't caught out by your proposed change. PEPs provide > that. (They also provide useful pointers for the "What's New" lists, > eg PEP 441.) > > So, are you proposing a change to Python? Then propose it. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Apr 5 06:00:49 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 5 Apr 2015 14:00:49 +1000 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: <20150405040049.GN25453@ando.pearwood.info> On Sat, Apr 04, 2015 at 04:16:47PM -0400, Alexander Belopolsky wrote: > I am genuinely interested in the ways to improve date/time formatting in > Python. There are certainly better ways than stftime. For example, ICU > has a date/time format syntax that is much more readable: instead of > "%Y-%m-%d %H:%M:%S", ICU format is "yyyy-MM-dd HH:mm:ss". > > I don't think it is hard to find a way to introduce ICU format in > datetime.__format__ while preserving backward compatibility. For example, > we may require that % is always "escaped" as '%' in ICU format > and the presence of unescaped % can trigger strftime interpretation. > > > [1] http://userguide.icu-project.org/formatparse/datetime That would be much more understandable to anyone who has used Excel or Excel-compatible spreadsheets. I wouldn't even *attempt* to add it to datetime.__format__, I would create either: - a new datetime method which uses ICU/Excel syntax directly: today = datetime.datetime.now().format_picture("yyyy-MM-dd") - or a new module function which converts ICU/Excel syntax to strftime format, which would allow post-processing in full generality: fmt = "Today is: " + datetime.make_format("yyyy-MM-dd") today = datetime.now().strftime(fmt, datetime.now()) Either is much more explicit and less magical than something which tries to guess whether a format string is ICU format or strftime format from the format string. But frankly, I would be satisfied with the list and meaning of format codes being documented somewhere where I can inspect it at runtime using help(), without having to run `man strftime` on a Linux system or search the Internet. Even something as trivial as giving the datetime module a list of format codes would work for me: print(datetime.CODES) -> [('%y', 'two digit year'), ('%Y', 'four digit year'), ... ] although having it pretty-print by default would be nice :-) As I understand it, the list of codes accepted is platform-dependent. While many codes are standard from machine to machine, some are not available on some systems. -- Steve From alexander.belopolsky at gmail.com Sun Apr 5 06:16:18 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sun, 5 Apr 2015 00:16:18 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <20150405040049.GN25453@ando.pearwood.info> References: <20150405040049.GN25453@ando.pearwood.info> Message-ID: On Sun, Apr 5, 2015 at 12:00 AM, Steven D'Aprano wrote: > But frankly, I would be satisfied with the list and meaning of format > codes being documented somewhere where I can inspect it at runtime using > help(), without having to run `man strftime` on a Linux system or search > the Internet. > That's easy. We added format codes to help('time.strftime') a couple years ago [1]. We can easily add the same to datetime.strftime. Feel free to open an issue. We don't need a PEP for that. :-) [1] http://bugs.python.org/issue9650 -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Apr 5 09:32:05 2015 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 05 Apr 2015 16:32:05 +0900 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: <87oan3f2tm.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Barker writes: > Note also that a PEP does not need to be the first step. Please do not feed the troll. Anatoly's technical ideas can be reasonably interesting, and as you see in the "datetime format" subthread, can trigger useful discussion. His ideas about workflow, however, are completely self-centered and inappropriate for Python. They do not deserve any more discussion in public. Already he has wasted many hours of several developers' time. You're welcome to try again, but please keep it off-list until you have some movement on his part to show. Nor does he ever share any content with the community; historically he has been a CLA refusenik. (To be fair, he may have changed in the interval since he was banned from the list and when the ban was lifted, but he hasn't said so yet.) Ironically enough, a PEP is about the only thing he's willing to provide the necessary legal authority to incorporate in Python! (At least, I don't recall being asked for a CLA to contribute to a PEP.) From breamoreboy at yahoo.co.uk Sun Apr 5 09:48:15 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 05 Apr 2015 08:48:15 +0100 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <87oan3f2tm.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87oan3f2tm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 05/04/2015 08:32, Stephen J. Turnbull wrote: > > Nor does he ever share any content with the community; historically he > has been a CLA refusenik. (To be fair, he may have changed in the > interval since he was banned from the list and when the ban was > lifted, but he hasn't said so yet.) Ironically enough, a PEP is about > the only thing he's willing to provide the necessary legal authority > to incorporate in Python! (At least, I don't recall being asked for a > CLA to contribute to a PEP.) > Rather difficult to have legal problems with the PEP copyright notice that always states "This document has been placed in the public domain.", or is that tempting fate? :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From steve at pearwood.info Sun Apr 5 12:09:20 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 5 Apr 2015 20:09:20 +1000 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: <87oan3f2tm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20150405100912.GQ25453@ando.pearwood.info> On Sun, Apr 05, 2015 at 08:48:15AM +0100, Mark Lawrence wrote: > Rather difficult to have legal problems with the PEP copyright notice > that always states "This document has been placed in the public > domain.", or is that tempting fate? :) According to the Creative Commons legal department, it is not always possible to attribute a work into the public domain. You can say it is public domain, but it isn't, it's still copyrighted, and even if you never enforce it, your heirs might. http://creativecommons.org/about/cc0 -- Steve From python at mrabarnett.plus.com Sun Apr 5 15:13:44 2015 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 05 Apr 2015 14:13:44 +0100 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: <55207392.8060608@mrabarnett.plus.com> Message-ID: <55213508.5080301@mrabarnett.plus.com> On 2015-04-05 01:45, Alexander Belopolsky wrote: > > On Sat, Apr 4, 2015 at 7:28 PM, MRAB > wrote: > > I am genuinely interested in the ways to improve date/time > formatting in > Python. There are certainly better ways than stftime. For > example, ICU > has a date/time format syntax that is much more readable: > instead of > "%Y-%m-%d %H:%M:%S", ICU format is "yyyy-MM-dd HH:mm:ss". > > I don't think it is hard to find a way to introduce ICU format in > datetime.__format__ while preserving backward compatibility. For > example, we may require that % is always "escaped" as '%' in > ICU format > and the presence of unescaped % can trigger strftime > interpretation. > > > [1] http://userguide.icu-project.org/formatparse/datetime > > Personally, I prefer some kind of escaping, possibly in the style of > .format, e.g. "{year}-{month}-{day} {hour}:{minute}:{second}". (It'll > probably need a little tinkering to shorten it! :-)) > > > Can someone explain to me why something like this or Anatoly's > double-curly-brace variant is an improvement over > > >>> from datetime import * > >>> "{0.year}-{0.month}-{0.day} > {0.hour}:{0.minute}:{0.second}".format(datetime.now()) > '2015-4-4 20:43:23' > Maybe it just needs the addition of attributes for '%A' and '%a' (name of day) and '%B' and '%b' (name of month). From abarnert at yahoo.com Mon Apr 6 01:57:16 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 5 Apr 2015 16:57:16 -0700 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <55213508.5080301@mrabarnett.plus.com> References: <55207392.8060608@mrabarnett.plus.com> <55213508.5080301@mrabarnett.plus.com> Message-ID: <8887F633-636E-4125-A8F6-84ED6D2F0F01@yahoo.com> On Apr 5, 2015, at 06:13, MRAB wrote: > >> On 2015-04-05 01:45, Alexander Belopolsky wrote: >> >> On Sat, Apr 4, 2015 at 7:28 PM, MRAB > wrote: >> >> I am genuinely interested in the ways to improve date/time >> formatting in >> Python. There are certainly better ways than stftime. For >> example, ICU >> has a date/time format syntax that is much more readable: >> instead of >> "%Y-%m-%d %H:%M:%S", ICU format is "yyyy-MM-dd HH:mm:ss". >> >> I don't think it is hard to find a way to introduce ICU format in >> datetime.__format__ while preserving backward compatibility. For >> example, we may require that % is always "escaped" as '%' in >> ICU format >> and the presence of unescaped % can trigger strftime >> interpretation. >> >> >> [1] http://userguide.icu-project.org/formatparse/datetime >> >> Personally, I prefer some kind of escaping, possibly in the style of >> .format, e.g. "{year}-{month}-{day} {hour}:{minute}:{second}". (It'll >> probably need a little tinkering to shorten it! :-)) >> >> >> Can someone explain to me why something like this or Anatoly's double-curly-brace variant is an improvement over >> >> >>> from datetime import * >> >>> "{0.year}-{0.month}-{0.day} {0.hour}:{0.minute}:{0.second}".format(datetime.now()) >> '2015-4-4 20:43:23' > Maybe it just needs the addition of attributes for '%A' and '%a' (name of day) and '%B' and '%b' (name of month). It probably needs most of the fields in strftime. While some of them may only be there because one guy at one Unix company needed it one time, things like 12-hour time and AM/PM, tz offset, month abbreviation, week number, etc. are not exactly rare, just a little uncommon; it would less than ideal to write most of your program with format, but have to fall back to strftime a couple times with a comment explaining that datetime is insufficient. (On the other hand, it is nice that you can do things like 0 prefix the same as any other format element, instead of needing to remember separate codes for with and without prefix.) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ericsnowcurrently at gmail.com Mon Apr 6 17:43:31 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 6 Apr 2015 09:43:31 -0600 Subject: [Python-ideas] Simpler Customization of Class Creation, next round In-Reply-To: References: Message-ID: On Tue, Mar 17, 2015 at 9:55 AM, Martin Teichmann wrote: > Hello everybody, > > recently I posted PEP 487, a simpler customization of class creation. > For newcomers: I propose the introduction of a __init_subclass__ classmethod > which initializes subclasses of a class, simplifying what metaclasses can > already do. >From what I've read, PEP 487 is adding functionality to classes that is already provided by metaclasses, but metaclasses still provide the identical capability. Basically, what is the difference between the following? class Meta(type): def __init__(cls, ...): ... class X(metaclass=Meta): ... and class Base: def __init_subclass__(cls, ...): ... class X(Base): ... The only difference I see is relative to metaclass conflcts. You still have to define an extra type (Meta or Base) to get the effect. Furthermore, there would now be second (but indistinct) way of initializing a subclass, making the "meta" class situation even murkier. If the only benefit is the mitigation of metaclass conflicts then perhaps such conflicts should be addressed directly and we should not add __init_subclass__. From other threads it sounds like we should be able to solve metaclass conflicts one way or another. Another thing I'm unclear on is what will happen if a subclass were to define its own __init_subclass__? -eric From random832 at fastmail.us Mon Apr 6 22:47:29 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 06 Apr 2015 16:47:29 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: References: Message-ID: <1428353249.3685694.249944689.09A09204@webmail.messagingengine.com> On Sat, Apr 4, 2015, at 00:18, Chris Angelico wrote: > There are many Pythons in the world. You can't just hack on CPython > and expect everything to follow on from there. Someone has to explain > to the Jython folks what they'll have to do to be compatible. Someone > has to write something up so MicroPython can run the same code that > CPython does. I thought that was the point of having pure python modules. If this can't be "figure it out, or use the pure python reference implementation already provided" then what's the point? From random832 at fastmail.us Mon Apr 6 22:58:11 2015 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 06 Apr 2015 16:58:11 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib Message-ID: <1428353891.3688072.249948785.783AA5C8@webmail.messagingengine.com> On Sun, Apr 5, 2015, at 00:00, Steven D'Aprano wrote: > As I understand it, the list of codes accepted is platform-dependent. > While many codes are standard from machine to machine, some are not > available on some systems. In my opinion all codes accepted by C11 *should* be supported by python; its present failure to do so is a wart. Any new work to address this issue should be in the form of a pure python strftime function rather than an effort to document every platform's idiosyncracies. In particular, none of the specifiers in the standard _except_ %z or %Z require any platform-specific logic. And these could be addressed by likewise supporting tm_zone and tm_gmtoff on platforms that do not provide them. From alexander.belopolsky at gmail.com Mon Apr 6 23:19:17 2015 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 6 Apr 2015 17:19:17 -0400 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <1428353891.3688072.249948785.783AA5C8@webmail.messagingengine.com> References: <1428353891.3688072.249948785.783AA5C8@webmail.messagingengine.com> Message-ID: On Mon, Apr 6, 2015 at 4:58 PM, wrote: > In my opinion all codes accepted by C11 *should* be supported by python; > its present failure to do so is a wart. Any new work to address this > issue should be in the form of a pure python strftime function rather > than an effort to document every platform's idiosyncracies. > An issue [1] to implement that was open almost 7 years ago and seen little progress since then. The problem is that there is no reliable cross-platform method to access locale information that stftime needs. One could try to do something with nl_langinfo(), but that is a pre-Unicode interface that assumes one global locale, so modern platforms have extended it in various incompatible ways. Note that strftime itself assumes one global locale and that's another reason to switch to something more modern. [1] http://bugs.python.org/issue3173 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ckaynor at zindagigames.com Mon Apr 6 23:40:02 2015 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Mon, 6 Apr 2015 14:40:02 -0700 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <1428353249.3685694.249944689.09A09204@webmail.messagingengine.com> References: <1428353249.3685694.249944689.09A09204@webmail.messagingengine.com> Message-ID: On Mon, Apr 6, 2015 at 1:47 PM, wrote: > On Sat, Apr 4, 2015, at 00:18, Chris Angelico wrote: >> There are many Pythons in the world. You can't just hack on CPython >> and expect everything to follow on from there. Someone has to explain >> to the Jython folks what they'll have to do to be compatible. Someone >> has to write something up so MicroPython can run the same code that >> CPython does. > > I thought that was the point of having pure python modules. If this > can't be "figure it out, or use the pure python reference implementation > already provided" then what's the point? Many of the other Python instances may have optimized versions of the libraries (CPython itself often does so, as does PyPy), or be using APIs from their base language (Jython and IronPython are good examples). In any of those cases, they have to update their versions/wrappers to support the new documented behavior. One of the points of a PEP is to document the changes so that those other versions know what they must change in their libraries, which may not be merely a port of the Python implementation. In fact, without a PEP system, it is likely that the changes could just fall under the rug and be completely missed, causing each implementation to have its effectively have its own standard library version, none of which are fully compatible with each other (and worse when may generally seem compatible). In addition to documenting the changes, the PEP process also allows the maintainers of other implementations (as well as the CPython core developers and anybody else interested) to officially provide feedback and concerns about the purposed changes. That all said, one of the common steps of PEP writing is to fork/branch the CPython code base and create a reference implementation of the proposal. The could be done merely by making a PyPi package or other third-party library for simpler changes or additions, or an actual fork/branch for more detailed or root changes (such as attempts to remove the GIL or changing syntax). Additionally, the pure python modules have other benefits. Namely, they can act as a fallback for when the optimized C modules cannot function, such as running on a platform which has not been fully optimized for. Chris From breamoreboy at yahoo.co.uk Tue Apr 7 00:16:33 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 06 Apr 2015 23:16:33 +0100 Subject: [Python-ideas] Top 10 Python modules that need a redesign Was: Geo coordinates conversion in stdlib In-Reply-To: <1428353891.3688072.249948785.783AA5C8@webmail.messagingengine.com> References: <1428353891.3688072.249948785.783AA5C8@webmail.messagingengine.com> Message-ID: On 06/04/2015 21:58, random832 at fastmail.us wrote: > On Sun, Apr 5, 2015, at 00:00, Steven D'Aprano wrote: >> As I understand it, the list of codes accepted is platform-dependent. >> While many codes are standard from machine to machine, some are not >> available on some systems. > > In my opinion all codes accepted by C11 *should* be supported by python; > its present failure to do so is a wart. Any new work to address this > issue should be in the form of a pure python strftime function rather > than an effort to document every platform's idiosyncracies. > > In particular, none of the specifiers in the standard _except_ %z or %Z > require any platform-specific logic. And these could be addressed by > likewise supporting tm_zone and tm_gmtoff on platforms that do not > provide them. Patches are always welcome. Will *YOU* be providing more patches than the originator of this thread? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From szymon at pythonista.net Tue Apr 7 16:54:51 2015 From: szymon at pythonista.net (=?UTF-8?B?U3p5bW9uIFB5xbxhbHNraQ==?=) Date: Tue, 07 Apr 2015 16:54:51 +0200 Subject: [Python-ideas] Pattern matching Message-ID: <5523EFBB.7080900@pythonista.net> Hello! I was trying to google if there were some attempts to add pattern matching feature (as seen in e.g. haskell) to Python. I couldn't however find anything. Were there any proposals for a feature like this? Greetings zefciu From toddrjen at gmail.com Tue Apr 7 17:44:52 2015 From: toddrjen at gmail.com (Todd) Date: Tue, 7 Apr 2015 17:44:52 +0200 Subject: [Python-ideas] Pattern matching In-Reply-To: <5523EFBB.7080900@pythonista.net> References: <5523EFBB.7080900@pythonista.net> Message-ID: On Tue, Apr 7, 2015 at 4:54 PM, Szymon Py?alski wrote: > Hello! > > I was trying to google if there were some attempts to add pattern > matching feature (as seen in e.g. haskell) to Python. I couldn't however > find anything. Were there any proposals for a feature like this? > Not everyone is familiar with Haskell. Can you provide an example of how this would work (in pseudocode, perhaps)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Apr 7 17:46:08 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Apr 2015 16:46:08 +0100 Subject: [Python-ideas] Pattern matching In-Reply-To: <5523EFBB.7080900@pythonista.net> References: <5523EFBB.7080900@pythonista.net> Message-ID: On 7 April 2015 at 15:54, Szymon Py?alski wrote: > I was trying to google if there were some attempts to add pattern > matching feature (as seen in e.g. haskell) to Python. I couldn't however > find anything. Were there any proposals for a feature like this? >From a quick look: https://pypi.python.org/pypi/patterns https://pypi.python.org/pypi/py-pattern-matching I'm not sure if either of these is the sort of thing you're looking for. Paul From guido at python.org Tue Apr 7 17:52:43 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Apr 2015 08:52:43 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: References: <5523EFBB.7080900@pythonista.net> Message-ID: Quick idea: this might integrate well with PEP 484, or with the (still speculative) multiple dispatch that Lukas Lang wants to write. On Apr 7, 2015 8:46 AM, "Paul Moore" wrote: > On 7 April 2015 at 15:54, Szymon Py?alski wrote: > > I was trying to google if there were some attempts to add pattern > > matching feature (as seen in e.g. haskell) to Python. I couldn't however > > find anything. Were there any proposals for a feature like this? > > From a quick look: > > https://pypi.python.org/pypi/patterns > https://pypi.python.org/pypi/py-pattern-matching > > I'm not sure if either of these is the sort of thing you're looking for. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From szymon at pythonista.net Tue Apr 7 21:25:21 2015 From: szymon at pythonista.net (=?UTF-8?B?U3p5bW9uIFB5xbxhbHNraQ==?=) Date: Tue, 07 Apr 2015 21:25:21 +0200 Subject: [Python-ideas] Pattern matching In-Reply-To: References: <5523EFBB.7080900@pythonista.net> Message-ID: <55242F21.8010002@pythonista.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07.04.2015 17:52, Guido van Rossum wrote: > > On Apr 7, 2015 8:46 AM, "Paul Moore" > wrote: > > On 7 April 2015 at 15:54, Szymon Py?alski > wrote: >> I was trying to google if there were some attempts to add >> pattern matching feature (as seen in e.g. haskell) to Python. I >> couldn't > however >> find anything. Were there any proposals for a feature like this? > > From a quick look: > > https://pypi.python.org/pypi/patterns > https://pypi.python.org/pypi/py-pattern-matching > Thanks these are nice but I think they try to mimic languages based on algebraic data types too much. I was thinking about something like this: Magic method for patterns - ----------------------------- Any object can be a pattern. If an object has a special method (called ``__pattern__`` or ``__match__``) defined then this method will govern how this object behaves as pattern. If not - pattern matching works the same as ``__eq__`` The ``__pattern__`` method can return ``False`` for no match. For a match it can return ``True`` (simplest patterns) a sequence or a mapping . Syntax for pattern matching - ------------------------------- The syntax could look something like this:: for object: pattern1: statement1 pattern2: statement2 statement3 pattern3 as var1, var2: statement4 statement5 For the simplest cases this works simply by calling appropriate statement block. If the ``__pattern__`` method returns a mapping, then the values from it will be merged into the local namespace. If it returns a sequence and there is the ``as`` clause, the values will be assigned to variables specified. > Quick idea: this might integrate well with PEP 484, or with the > (still speculative) multiple dispatch that Lukas Lang wants to > write. Function dispatch - -----------------------[ Besides the above syntax the patterns could be also used to register functions for dispatching. I dont' know how precisely it would integrate with PEP 484, but I agree that any valid type hint should also be a valid pattern. Existing objects as patterns - ---------------------------------- * Types should match their instances * Tuples should match sequences that have the same length and whose elements match against the elements of a tuple (subpatterns) * Dictionaries should match mappings that contain all the keys in the pattern and the values match the subpatterns under these keys * Regular expressions should match strings. For named groups they should populate the local namespace. * The ``Ellipsis`` object can be used as a match-all pattern. Example - ----------- The code below would work if the ``point`` variable is specified as: * a tuple (x, y) * a mapping {'x': x, 'y': y} * a string "{x}-{y}".format(x, y) :: def print_point(x, y): print('x = {}'.format(x)) print('y = {}'.format(y)) for point: (Number, Number) as x, y: print_point(x, y) {'x': Number, 'y': Number}: print_point(x, y) re.compile('(?P[0-9]+)-(?P[0-9])+'): # Or re.compile('([0-9]+)-([0-9])+') as x, y: print_point(int(x), int(y)) ...: raise TypeError('Expected something else') Greetings zefciu -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJVJC8YAAoJEBoIz6vZoBBhm04H/i5dJAMTloRxqSXXyY7QU8kx 0gBPjCoCsw6XZSu1nlKP0/yY+tO5uXjQEK5s1vid8gjuEBnrZCtwjfCYiiqSs96P 7iT9kSnyx/0jr5FvAC1ZItb3KN7G+MBxMfCHENM8yMz7Yxdw2F5ziT7zFE5aOELW y3Mz37+GytmMUT/DXA4wIHBFJgALspWgoU2P/ilzg4oCtbQREmX5pcMnFIsMavPI oIu7KJ0ZW2hJ4K2a0h8HWCpZ9aYwcCTF51HMkTAXo6leLSm3XG36jKiCEFJiZpxe sLH8e/2rF93tBUVunJfyEQG/bOnlkoTX2rONavDZCqSqt8t3ehT170TtHJL10Os= =iO75 -----END PGP SIGNATURE----- From rosuav at gmail.com Tue Apr 7 21:52:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Apr 2015 05:52:23 +1000 Subject: [Python-ideas] Pattern matching In-Reply-To: <55242F21.8010002@pythonista.net> References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> Message-ID: On Wed, Apr 8, 2015 at 5:25 AM, Szymon Py?alski wrote: > Syntax for pattern matching > - ------------------------------- > > The syntax could look something like this:: > > for object: > pattern1: statement1 > pattern2: > statement2 > statement3 > pattern3 as var1, var2: > statement4 > statement5 > > For the simplest cases this works simply by calling appropriate > statement block. If the ``__pattern__`` method returns a mapping, then > the values from it will be merged into the local namespace. If it > returns a sequence and there is the ``as`` clause, the values will be > assigned to variables specified. So, tell me if I'm understanding you correctly: The advantage over a basic if/elif/else tree is that it can assign stuff as well as giving a true/false result? Because if all you want is the true/false, there's not a lot of advantage over what currently exists. The sequence and "as" clause syntax is reasonable, but I do not like the idea that a mapping's keys would quietly become local name bindings. It'd be like "from ... import *" inside a function - suddenly _any_ name could become local, without any way for the compiler to know. Also - although this is really just bikeshedding - not a fan of the use of 'for' here. I'm imagining code something like this: for value in gen(): # this one iterates for value: # this one doesn't int: yield ("(%d)" if value<0 else "%d") % value str: yield repr(value) datetime.datetime: yield value.strftime(timefmt) ...: yield str(value) Come to think of it, not a fan of ellipsis here either; "else" seems better. But that's minor bikeshedding too. The key is to show how this is materially better than an if/elif tree *and* better than a dictionary lookup. (Obviously isinstance checks are superior to dict lookups based on type(value), but I don't know how often people actually write code like the above.) > Besides the above syntax the patterns could be also used to register > functions for dispatching. I dont' know how precisely it would > integrate with PEP 484, but I agree that any valid type hint should > also be a valid pattern. > > Existing objects as patterns > - ---------------------------------- > > * Types should match their instances > * Tuples should match sequences that have the same length and whose > elements match against the elements of a tuple (subpatterns) > * Dictionaries should match mappings that contain all the keys in the > pattern and the values match the subpatterns under these keys > * Regular expressions should match strings. For named groups they > should populate the local namespace. > * The ``Ellipsis`` object can be used as a match-all pattern. This is where the type hints could save you a lot of trouble. They already define a sloppy form of isinstance checking, so you don't need to do all the work of defining tuples, dicts, etc - all you have to say is "typing.py type hint objects match any object which they would accept", and let the details be handled elsewhere. That'd get you Union types and such, as well as what you're talking about above. > for point: > (Number, Number) as x, y: > print_point(x, y) > {'x': Number, 'y': Number}: > print_point(x, y) > re.compile('(?P[0-9]+)-(?P[0-9])+'): > # Or re.compile('([0-9]+)-([0-9])+') as x, y: > print_point(int(x), int(y)) > ...: > raise TypeError('Expected something else') I'm not sure how to rescue the "regex with named groups" concept, but this code really doesn't look good. Using "as" wouldn't work reliably, since dicts are iterable (without a useful order). Something along the lines of: x, y from re.compile('(?P[0-9]+)-(?P[0-9])+'): or maybe switch the arguments (to be more like "from ... import x, y")? But whatever it is, I'd prefer to be explicit: This has to return a dictionary containing keys 'x' and 'y' (what if there are others? Ignore them?), and then they will be bound to identical names. All in all, an interesting idea, but one that's going to require a good bit of getting-the-head-around, so it wants some strong use cases. Got some real-world example code that would benefit from this? ChrisA From abarnert at yahoo.com Tue Apr 7 23:10:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 7 Apr 2015 14:10:46 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: <5523EFBB.7080900@pythonista.net> References: <5523EFBB.7080900@pythonista.net> Message-ID: <985AE3FB-240B-4DCE-8B9A-E3C9421480A9@yahoo.com> On Apr 7, 2015, at 07:54, Szymon Py?alski wrote: > > Hello! > > I was trying to google if there were some attempts to add pattern > matching feature (as seen in e.g. haskell) to Python. I couldn't however > find anything. Were there any proposals for a feature like this? http://stupidpythonideas.blogspot.com/2014/08/a-pattern-matching-case-statement-for.html I wrote this up after getting frustrated with people suggesting Python should have "a case statement like every other language" where by "every other language" they meant C and Pascal and their descendants with the relatively useless case that's just an elif chain rather than ML and its descendants with the more useful pattern matching case statement. (Nick Coghlan pointed out that go has an interesting middle ground--it looks like a JavaScript switch, but with an as clause to optionally bind the result of each expression to a local variable.) Once you've defined the case statement, you can add pattern-matching assignments and function definitions (especially if you know Haskell, where the equivalents are just syntactic sugar for using case). The big problem that you have in Python that you don't have in ML, Haskell, etc. is that most values don't have a unique, evaluable representation that can be pattern-matched. I think the best solution to this is a __match__ protocol, but I went over some other ideas. The smaller problem is that in Python, only functions (and classes and modules) have scope; the idea of a name bound "locally" is tricky. There's a bit of hackery around except clauses, and comprehensions are expressions that get compiled into calls to hidden functions in part to deal with this, but I'm not sure either of those is something you'd want to extend here. Anyway, I think that my straw man proposal is a start toward the best design you'd be able to come up with (although it's not in itself the best design, probably), and it still doesn't fit into Python. But read it and tell me if you disagree with either half of that. > Greetings > zefciu > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Wed Apr 8 00:59:44 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 7 Apr 2015 15:59:44 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: <55242F21.8010002@pythonista.net> References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> Message-ID: <9CA5FBB2-878A-4859-BEA1-9D28AFE5C61C@yahoo.com> On Apr 7, 2015, at 12:25, Szymon Py?alski wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > >> On 07.04.2015 17:52, Guido van Rossum wrote: >> >> On Apr 7, 2015 8:46 AM, "Paul Moore" > > wrote: >> >> On 7 April 2015 at 15:54, Szymon Py?alski > > wrote: >>> I was trying to google if there were some attempts to add >>> pattern matching feature (as seen in e.g. haskell) to Python. I >>> couldn't >> however >>> find anything. Were there any proposals for a feature like this? >> >> From a quick look: >> >> https://pypi.python.org/pypi/patterns >> https://pypi.python.org/pypi/py-pattern-matching > Thanks these are nice but I think they try to mimic languages based on > algebraic data types too much. I was thinking about something like this: > > Magic method for patterns > - ----------------------------- > > Any object can be a pattern. If an object has a special method (called > ``__pattern__`` or ``__match__``) defined then this method will govern > how this object behaves as pattern. If not - pattern matching works > the same as ``__eq__`` "Works the same as __eq__" either doesn't allow you to assign parts while decomposing, or turns out to have a lot of problems. In my proposal, I restricted that to only work on values that are expressions that can't be read as assignment targets, and handled assignment lists recursively, to get around those problems. Yours just punts on decomposition here and offers something different (the mapping in the next paragraph), but I don't think that covers many common cases. > The ``__pattern__`` method can return ``False`` for no match. For a > match it can return ``True`` (simplest patterns) a sequence or a mapping > . The mapping is an interesting end-run around the decomposition problem, but it seems like it throws away the type of the object. In other words, Point(x, y) and Vector(x, y) both pattern-match identically to either {'x': x, 'y': y}. In ML or Haskell, distinguishing between the two is one of the key uses of pattern matching. And another key use is distinguishing Point(0, y), Point(x, 0) and Point(x, y), which it looks like you also can't do this way; the only way to get x bound in the case block (assuming Point, like most classes, isn't iterable and therefore can't be decomposed as a tuple) is to match a dict. > Syntax for pattern matching > - ------------------------------- > > The syntax could look something like this:: > > for object: > pattern1: statement1 > pattern2: > statement2 > statement3 > pattern3 as var1, var2: > statement4 > statement5 I don't like "for" here. Avoiding new keywords is good, but reusing a keyword to mean something unrelated is often confusing. With "for object in iterable:", the "object" names the thing that will be assigned each time through the loop. Here, it looks like you're omitting the "in iterable", but really you're omitting the "object in" part, which is half of one clause and half of another. Also, in cases where the matchable object/iterable is a long and complex expression (as it often is in both pattern matching and loops), it'll be very confusing to the eye if you misinterpret which kind of for you have. On the other hand, I like the keyword-less match clauses better than my version with "of". The syntax may be harder to describe to the grammar, but it's easier to a read for a human. The only potential advantage I can see to the "of" clause is that it opens the door to let a single-match case be written without requiring a double indent, or even in a single line: case obj: of pattern: statement statement case obj: of pattern: statement But I left that out of my proposal because it breaks the compound statement rule that (informally) you can't put two colons on the same line. > For the simplest cases this works simply by calling appropriate > statement block. If the ``__pattern__`` method returns a mapping, then > the values from it will be merged into the local namespace. If it > returns a sequence and there is the ``as`` clause, the values will be > assigned to variables specified. > >> Quick idea: this might integrate well with PEP 484, or with the >> (still speculative) multiple dispatch that Lukas Lang wants to >> write. > > Function dispatch > - -----------------------[ > > Besides the above syntax the patterns could be also used to register > functions for dispatching. I dont' know how precisely it would > integrate with PEP 484, but I agree that any valid type hint should > also be a valid pattern. Consider the syntax from Matthew Rocklin's multipledispatch library on PyPI: @dispatch(int, int) def add(x, y): return x+y @dispatch(object, object): def add(x, y): return "%s + %s" % (x, y) Lukas pointed out that you can out the types in the annotations rather than the decorator, and use MyPy syntax for them instead of having to invent a similar but not identical syntax as Matthew does. Anyway, it should be easy to see how this maps to a single function with pattern matching and vice-versa. Especially once you look at how Haskell implements overloading--it's just syntactic sugar for a single definition with a case expression inside. > Existing objects as patterns > - ---------------------------------- > > * Types should match their instances This is an interesting idea. This allows you to do something halfway between taking any value and checking a specific value--and something that would often be useful in Python--which my proposal didn't have. If we had a syntax for annotating values or variables (rather than just function params) with types, you could do the same thing by specifying "_: Number", which means you could also match "x: Number" to check the type and bind the value. That's something that was missing from my proposal (see the JSON schema example in the linked post on ADTs). Without that, the only way I can think of to check the type but nothing else in my proposal is something like Breakfast(*) or something equally ugly, so that's a win for your syntax. > * Tuples should match sequences that have the same length and whose > elements match against the elements of a tuple (subpatterns) My proposal just used the same rules as assignment to a target list (except that it recurses on pattern matching each element instead of assigning each element, of course). The upside is more flexibility, and the details have already been worked out and proven useful in Python assignments (and extended a couple of times over the past few decades in ways that work just as well here as they did there). The downside is that if you pattern match an iterator, each tuple decomposition case will eat elements from the iterator. But presumably anyone who understands iterators already knows that. > * Dictionaries should match mappings that contain all the keys in the > pattern and the values match the subpatterns under these keys This definitely solves the problem of how to deal with objects that are usually constructed with keyword arguments. But it doesn't help for objects usually constructed with a combination of positional and keyword arguments. (Of course you can make __pattern__ return an OrderedDict; the problem is on the other end, how to write a pattern that matches arg #0, arg #1, and arg "spam" all in one pattern.) Anyway, I still don't like the idea of neither getting nor matching the type here, or the idea of dumping all the names into locals instead of binding names that you ask for specifically, as I mentioned earlier. Also, consider how you'd map this relatively common ML/Haskell use: case p1 of Point(x1, y1): case p2 of Point(x2, y2): if x1 <= x < x2 and y1 <= y < y2: With a syntax like mine, where you specify the names to be bound, this is easy; with yours, where the pattern just dumps its names into the namespace, you have to write something like: x0, y0 = x, y case p1 of {'x': Any, 'y': Any}: x1, y1 = x, y case p2 of {'x': Any, 'y': Any}: x2, y2 = x, y if x1 <= x0 < x2 and y1 <= y0 < y2 > * Regular expressions should match strings. For named groups they > should populate the local namespace. Good idea. Anyone coming from another modern scripting language would expect that. Then again, they expect regexp support in a whole lot of places Python doesn't have them, and that may not be a bad thing. Code that's inherently about regexps always looks simpler in other languages than in Python--but code that really isn't about regexps often gets written with regexps anyway in those languages, but rethought into something more readable in Python... And here, my alternative of specifying values to check and names to bind instead of just dumping everything into the local namespace doesn't seem to fit very well. (Without regexp literals as part of the language syntax instead of as strings, you can't cram an arbitrary target or expression inside the

block.) > * The ``Ellipsis`` object can be used as a match-all pattern. I used the else keyword, but if you're going with keywordless match clauses, this makes sense. > Example > - ----------- > > The code below would work if the ``point`` variable is specified as: > > * a tuple (x, y) > * a mapping {'x': x, 'y': y} > * a string "{x}-{y}".format(x, y) But what if it's specified as an instance of a Point type, which is pretty much the paradigm case for pattern matching, and the only one that causes a problem? Also, this raises an obvious problem with the regexp alternative: the first two cases match any Number, but the last case only matches nonnegative integers. And think about what it would mean to match any Number instance with a regexp. It's not just impossible, it doesn't even make sense, because values and their reprs aren't the same thing. If I import quaternion, its instances will match for the first two, but their reprs definitely won't match your regexp. This is one of those "now they have two problems" cases: a regexp looks like it will give you a simple solution to what you want, but it's just misleading you into thinking there's a simple solution that won't even pass your second test case. > :: > > def print_point(x, y): > print('x = {}'.format(x)) > print('y = {}'.format(y)) > > for point: > (Number, Number) as x, y: > print_point(x, y) > {'x': Number, 'y': Number}: > print_point(x, y) > re.compile('(?P[0-9]+)-(?P[0-9])+'): > # Or re.compile('([0-9]+)-([0-9])+') as x, y: > print_point(int(x), int(y)) > ...: > raise TypeError('Expected something else') > > > Greetings > zefciu > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1 > > iQEcBAEBAgAGBQJVJC8YAAoJEBoIz6vZoBBhm04H/i5dJAMTloRxqSXXyY7QU8kx > 0gBPjCoCsw6XZSu1nlKP0/yY+tO5uXjQEK5s1vid8gjuEBnrZCtwjfCYiiqSs96P > 7iT9kSnyx/0jr5FvAC1ZItb3KN7G+MBxMfCHENM8yMz7Yxdw2F5ziT7zFE5aOELW > y3Mz37+GytmMUT/DXA4wIHBFJgALspWgoU2P/ilzg4oCtbQREmX5pcMnFIsMavPI > oIu7KJ0ZW2hJ4K2a0h8HWCpZ9aYwcCTF51HMkTAXo6leLSm3XG36jKiCEFJiZpxe > sLH8e/2rF93tBUVunJfyEQG/bOnlkoTX2rONavDZCqSqt8t3ehT170TtHJL10Os= > =iO75 > -----END PGP SIGNATURE----- > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From grant.jenks at gmail.com Wed Apr 8 01:35:07 2015 From: grant.jenks at gmail.com (Grant Jenks) Date: Tue, 7 Apr 2015 16:35:07 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: <9CA5FBB2-878A-4859-BEA1-9D28AFE5C61C@yahoo.com> References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> <9CA5FBB2-878A-4859-BEA1-9D28AFE5C61C@yahoo.com> Message-ID: On Tue, Apr 7, 2015 at 3:59 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > The mapping is an interesting end-run around the decomposition problem, > but it seems like it throws away the type of the object. In other words, > Point(x, y) and Vector(x, y) both pattern-match identically to either {'x': > x, 'y': y}. In ML or Haskell, distinguishing between the two is one of the > key uses of pattern matching. > > And another key use is distinguishing Point(0, y), Point(x, 0) and > Point(x, y), which it looks like you also can't do this way; the only way > to get x bound in the case block (assuming Point, like most classes, isn't > iterable and therefore can't be decomposed as a tuple) is to match a dict. > Wouldn't it be better to be explicit about the type matching? For example, using pypatt, I've done this: In [19]: import pypatt In [20]: Point = namedtuple('Point', 'x y') In [21]: @pypatt.transform def origin_distance(point): with match((type(point), point)): with (Point, (0, quote(y))): return y with (Point, (quote(x), 0)): return x with (Point, (quote(x), quote(y))): return (x ** 2 + y ** 2) ** 0.5 ....: In [22]: origin_distance(Point(0, 5)) Out[22]: 5 In [23]: origin_distance(Point(10, 0)) Out[23]: 10 In [24]: origin_distance(Point(3, 4)) Out[24]: 5.0 In [25]: origin_distance(Point(0, 'far')) Out[25]: 'far' In [26]: origin_distance(Point('near', 0)) Out[26]: 'near' https://pypi.python.org/pypi/pypatt/ Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Apr 8 03:10:42 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 7 Apr 2015 18:10:42 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> <9CA5FBB2-878A-4859-BEA1-9D28AFE5C61C@yahoo.com> Message-ID: On Apr 7, 2015, at 16:35, Grant Jenks wrote: > >> On Tue, Apr 7, 2015 at 3:59 PM, Andrew Barnert wrote: >> The mapping is an interesting end-run around the decomposition problem, but it seems like it throws away the type of the object. In other words, Point(x, y) and Vector(x, y) both pattern-match identically to either {'x': x, 'y': y}. In ML or Haskell, distinguishing between the two is one of the key uses of pattern matching. >> >> And another key use is distinguishing Point(0, y), Point(x, 0) and Point(x, y), which it looks like you also can't do this way; the only way to get x bound in the case block (assuming Point, like most classes, isn't iterable and therefore can't be decomposed as a tuple) is to match a dict. > > Wouldn't it be better to be explicit about the type matching? You mean matching the type of the object? If so, that's my point. A Point and a Vector aren't the same thing just because they have the same two members, which is why in ML or Haskell or my proposal you have to match Point(x, y), not just (x, y) or {'x': x, 'y': y}. (If you mean the types of the matched elements, the trick is to be able to specify the name and the type in one place; the obvious solution is to extend annotations to assignment/match targets, so you can match Point(x: int, y: int). But anyway, I don't think that's what you meant.) With a library that doesn't extend Python syntax, that would have to look something like (Point, (q.x, q.y)), which you can probably make a little nicer with some hacks that inspect a compiled function object or a stack frame (to remove the need for quotes around variable names), but then presumably you knew that because you wrote pypatt, which I'm guessing uses one of those hacks. :) > For example, using pypatt, I've done this: > > In [19]: import pypatt > > In [20]: Point = namedtuple('Point', 'x y') > > In [21]: @pypatt.transform > def origin_distance(point): > with match((type(point), point)): > with (Point, (0, quote(y))): > return y > with (Point, (quote(x), 0)): > return x > with (Point, (quote(x), quote(y))): > return (x ** 2 + y ** 2) ** 0.5 > ....: In my proposed syntax, you'd write that as: case point: of Point(0, y): return y of Point(x, 0): return x of Point(x, y): return (x ** 2 + y ** 2) ** 0.5 In his syntax, I don't think you can write it at all. You either get a type match or a decomposition, not both, and there's no way (implicitly or explicitly) to match targets to bind and values to check; the best you can do is something like: for point: Point: if x == 0: return y elif y == 0: etc. ... using the fact that Point.__pattern__ returns a dict which is magically dumped into the enclosing scope. So it's simultaneously less explicit and more verbose. > In [22]: origin_distance(Point(0, 5)) > Out[22]: 5 > > In [23]: origin_distance(Point(10, 0)) > Out[23]: 10 > > In [24]: origin_distance(Point(3, 4)) > Out[24]: 5.0 > > In [25]: origin_distance(Point(0, 'far')) > Out[25]: 'far' > > In [26]: origin_distance(Point('near', 0)) > Out[26]: 'near' > > https://pypi.python.org/pypi/pypatt/ PyPatt looks pretty neat, but I suspect it has a _different_ fundamental problem. As far as I can tell from a quick glance, it doesn't seem to have any way for types to opt in to the matching protocol like Szymon's proposal or mine, which presumably means that it matches by constructing a value and checking it for equality, which means that any type that does anything non-trivial on construction is unmatchable (or maybe even dangerous--consider FileIO(path, 'w') as a pattern). I'm sure Szymon didn't want to add his __pattern__ protocol any more than I wanted to add my __match__ protocol, but once you start trying it with real types I don't think there's any real alternative. (See my linked blog post for more on that.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Apr 8 04:04:03 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Apr 2015 12:04:03 +1000 Subject: [Python-ideas] Pattern matching In-Reply-To: <985AE3FB-240B-4DCE-8B9A-E3C9421480A9@yahoo.com> References: <5523EFBB.7080900@pythonista.net> <985AE3FB-240B-4DCE-8B9A-E3C9421480A9@yahoo.com> Message-ID: On Wed, Apr 8, 2015 at 7:10 AM, Andrew Barnert wrote: > The smaller problem is that in Python, only functions (and classes and modules) have scope; the idea of a name bound "locally" is tricky. There's a bit of hackery around except clauses... > FWIW it's not hackery around scope at all - it's simply that the name gets unbound: >>> e = 2.71828 >>> try: 1/0 ... except ZeroDivisionError as e: print("1/0!") ... 1/0! >>> e Traceback (most recent call last): File "", line 1, in NameError: name 'e' is not defined ChrisA From abarnert at yahoo.com Wed Apr 8 04:45:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 7 Apr 2015 19:45:46 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: References: <5523EFBB.7080900@pythonista.net> <985AE3FB-240B-4DCE-8B9A-E3C9421480A9@yahoo.com> Message-ID: <789DA636-7D1F-4E87-A320-9181CD1E5CF0@yahoo.com> On Apr 7, 2015, at 19:04, Chris Angelico wrote: > > On Wed, Apr 8, 2015 at 7:10 AM, Andrew Barnert > wrote: >> The smaller problem is that in Python, only functions (and classes and modules) have scope; the idea of a name bound "locally" is tricky. There's a bit of hackery around except clauses... > > FWIW it's not hackery around scope at all - it's simply that the name > gets unbound: Sorry, you're right, "hackery" is misleading here; I should have just said "(yes, I know about comprehensions and except clauses, but they're not relevant here)" or something similar. I don't consider the 3.x rules for except "hacky" or bad in any way, and didn't mean to imply otherwise. Anyway, an ML/Haskell case expression binds the matched variables inside the case branch; the most obvious interpretation of this proposal (or mine) would bind them in the entire function. Of course it's possible to solve that (if you think it needs solving) either the way comprehensions do (compile the case statement, or each branch, as a function definition and call) or the way except does (unbind names that appear in the "as" clause after the statement), or probably other ways, but if you don't do anything, they work like assignments. (Which is actually a perfectly good parallel to ML, where they work like let expressions, it's just that let expressions don't work like assignments.) From rosuav at gmail.com Wed Apr 8 06:56:45 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Apr 2015 14:56:45 +1000 Subject: [Python-ideas] Pattern matching In-Reply-To: <789DA636-7D1F-4E87-A320-9181CD1E5CF0@yahoo.com> References: <5523EFBB.7080900@pythonista.net> <985AE3FB-240B-4DCE-8B9A-E3C9421480A9@yahoo.com> <789DA636-7D1F-4E87-A320-9181CD1E5CF0@yahoo.com> Message-ID: On Wed, Apr 8, 2015 at 12:45 PM, Andrew Barnert wrote: > On Apr 7, 2015, at 19:04, Chris Angelico wrote: >> >> On Wed, Apr 8, 2015 at 7:10 AM, Andrew Barnert >> wrote: >>> The smaller problem is that in Python, only functions (and classes and modules) have scope; the idea of a name bound "locally" is tricky. There's a bit of hackery around except clauses... >> >> FWIW it's not hackery around scope at all - it's simply that the name >> gets unbound: > > Sorry, you're right, "hackery" is misleading here; I should have just said "(yes, I know about comprehensions and except clauses, but they're not relevant here)" or something similar. I don't consider the 3.x rules for except "hacky" or bad in any way, and didn't mean to imply otherwise. > Gotcha. > Anyway, an ML/Haskell case expression binds the matched variables inside the case branch; the most obvious interpretation of this proposal (or mine) would bind them in the entire function. > Of course it's possible to solve that (if you think it needs solving) either the way comprehensions do (compile the case statement, or each branch, as a function definition and call) or the way except does (unbind names that appear in the "as" clause after the statement), or probably other ways, but if you don't do anything, they work like assignments. (Which is actually a perfectly good parallel to ML, where they work like let expressions, it's just that let expressions don't work like assignments.) > I'd treat these case branches like 'with' statements. The name binding from "as" lasts past the end of the block, it's no different from any other assignment. The only reason exceptions unbind is to avoid the refloop of an exception having a reference to locals, and the local name referencing the exception. Everything else can be just like assignment. So basically, what we have here is a cross between a 'with' statement and an 'if'. I started writing up a demo that mainly used 'with', and then realized that I had no idea how to have the __enter__ method stipulate that the body should be skipped... oh, hello PEP 377, that's where the problem is. This proposal would need some new syntax, but conceptually, it'd be something like an __enter__ method returning "hey, this body should be skipped", based on whatever criteria it likes (isinstance, regex matching, etc). ChrisA From aj at erisian.com.au Wed Apr 8 07:12:43 2015 From: aj at erisian.com.au (Anthony Towns) Date: Wed, 8 Apr 2015 15:12:43 +1000 Subject: [Python-ideas] Pattern matching In-Reply-To: <55242F21.8010002@pythonista.net> References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> Message-ID: On 8 April 2015 at 05:25, Szymon Py?alski wrote: > Any object can be a pattern. If an object has a special method (called > ``__pattern__`` or ``__match__``) defined then this method will govern > how this object behaves as pattern. If not - pattern matching works > the same as ``__eq__`` > > The ``__pattern__`` method can return ``False`` for no match. For a > match it can return ``True`` (simplest patterns) a sequence or a mapping > ?Always returning a sequence or None would seem simpler to deal with, wouldn't it? I'm imaginging the default implementations would be something like: class object: def __match__(self, item): if item == self?: return [item] else: return None class type: def __match__(cls, item): if isinstance(item, cls): return [item] else: return None class tuple: def __match__(self, item): if isinstance(item, tuple): if all( si.__match__(ii) for si,ii in zip(self, item) ): return item return None Syntax for pattern matching > - ------------------------------- > The syntax could look something like this:: > > for object: > pattern1: statement1 > Is the two level indent actually a win? It saves re-evaluating the expression, but you could just as easly have: object = f(x, y+w, g(z/3 for z in x)) match object ~~ pattern1 as a,b,c: statements... ormatch object ~~ pattern2 as a,d: statements... ?but in that case you're coming pretty close to an 'if' block that's able to set locals: object = ... if object ~~ pattern1 as a,b,c: ... elif object ~~ pattern2 as a,d: ... (Using ~~ as a placeholder for the syntax for "matches against") for point: > (Number, Number) as x, y: > print_point(x, y) > {'x': Number, 'y': Number}: > print_point(x, y) > re.compile( > ?? > '(?P[0-9]+)-(?P[0-9])+'): > # Or re.compile('([0-9]+)-([0-9])+') as x, y: > print_point(int(x), int(y)) > ...: > raise TypeError('Expected something else') > My understanding of the way Haskell does name assignment in case matching is that it lets you put placeholders in a constructor. So where you might say x = MyX(a=1, b=2) to create an object 'x', you could then say something along the lines of: x ~~ MyX(a=foo, b=bar) to pull the values (1, 2) back out (as foo and bar). Doing exactly that doesn't make sense for python because reversing a constructor doesn't make much sense. If you could make that work, you'd get the python version of Haskell pattern matching looking like: re_pt = re.compile(?'(?P[0-9]+)-(?P[0-9])+') case point of: (Number(x), Number(y)): ... ('x': Number(x), 'y': Number(y)): ... re_pt(x=x, y=y): .. ?where you're at least mentioning the 'x' and 'y' variables that are getting filled in. Okay, probably horrible idea: Matching syntax looks like: case point is: (x:=Number, y:=Number): ... a:=x is the same as calling assigning_match(locals(), x, "a") run in the local scope. a,b:=x is the same as calling assigning_match(locals(), x, ("a", "b")). 'case obj is: \n pattern[0]: stmt[0] \n pattern[1]: stmt[1]' does: for i in range(2): try: pattern[i].__match__(obj) except NoMatch: continue stmt[i] break __match__() raises NoMatch on failure, or returns the match: class object: def __match__(self, item): if self == item: return item raise NoMatch class type: def __match__(self, item): if isinstance(item, self): return item raise NoMatch class tuple: def __match__(self, item): if isinstance(item, tuple): if all( s_i.__match__(i_i) for s_i, i_i in zip(self, item) ): return item raise NoMatch class dict: def __match__(self, item): if isinstance(item, dict): if item.keys() == self.keys(): if all(s_v.__match__(item[s_k]) for s_k,s_v in self.items()): return item raise NoMatch ? class SRE_Pattern: def __match__(self, item): m = self.match(it?em) if m is None: raise NoMatch return m.groups() ?assigning_match works something like:? class assigning_match(object): def __init__(self, l, matchobj, varname): assert isinstance(varname, str) or isinstance(varname, tuple) self.l = l self.m = matchobj self.n = varname def __match__(self, item): r = self.m.__match__(item) if isinstance(self.n, str): self.l[self.n] = other else: for n, r_i in zip(self.n, r): self.l[n] = r_i return r Then the example looks like: case point is: (x:=Number, y:=Number): ... {"x": x:=Number, "y": y:=Number}: ... x,y:=re_pt: ... Being able to pull out and name something that's a deep part of an object seems like it could be useful; otherwise it's not adding much over if/elif. Though maybe it'd be interesting to have a "deconstructor" protocol for that instead, ie: Foo(a,b,key=c) = foo is equivalent to something like: x = Foo.__deconstruct__(foo) a = x[0] b = x[1] c = x["key"] I think it might even extend sanely to unpacking subobjects: Foo(Bar(a), b) = foo becomes: x = Foo.__deconstruct__(foo) Bar(a) = x[0] b = x[1] becomes: x = Foo.__deconstruct__(foo) x2 = Bar.__deconstruct__(x[0]) a = x2[0] b = x[1] Using "_" to accept and ignore any value would work fine there. If you allowed literals instead of an identifier, you could treat: Foo(0, 99, key="x") = foo as: x = Foo.__deconstruct__(foo) assert 0 == x[0] assert 99 == x[1] assert "x" = x["key"] That'd be enough to use it for matching I think: case foo is: Foo(a,b): ... would become: try: Foo(a,b) = foo except: continue ... break But it might be confusing figuring out whether: b = 0 case foo is: Foo(a,b): print(a) should result in value checking: if foo ~~ Foo(a,0): print(a) or value assignment: del b if foo ~~ Foo(a,b): print(a) Eh, I don't know. Maybe something there is interesting though. Cheers, aj -- Anthony Towns -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Apr 8 08:39:36 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 7 Apr 2015 23:39:36 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: References: <5523EFBB.7080900@pythonista.net> <985AE3FB-240B-4DCE-8B9A-E3C9421480A9@yahoo.com> <789DA636-7D1F-4E87-A320-9181CD1E5CF0@yahoo.com> Message-ID: <29F1D83F-54CF-47B0-8213-DF3E96130ACA@yahoo.com> On Apr 7, 2015, at 21:56, Chris Angelico wrote: > >> On Wed, Apr 8, 2015 at 12:45 PM, Andrew Barnert wrote: >>> On Apr 7, 2015, at 19:04, Chris Angelico wrote: >>> >>> On Wed, Apr 8, 2015 at 7:10 AM, Andrew Barnert >>> wrote: >>>> The smaller problem is that in Python, only functions (and classes and modules) have scope; the idea of a name bound "locally" is tricky. There's a bit of hackery around except clauses... >>> >>> FWIW it's not hackery around scope at all - it's simply that the name >>> gets unbound: >> >> Sorry, you're right, "hackery" is misleading here; I should have just said "(yes, I know about comprehensions and except clauses, but they're not relevant here)" or something similar. I don't consider the 3.x rules for except "hacky" or bad in any way, and didn't mean to imply otherwise. > > Gotcha. > >> Anyway, an ML/Haskell case expression binds the matched variables inside the case branch; the most obvious interpretation of this proposal (or mine) would bind them in the entire function. >> Of course it's possible to solve that (if you think it needs solving) either the way comprehensions do (compile the case statement, or each branch, as a function definition and call) or the way except does (unbind names that appear in the "as" clause after the statement), or probably other ways, but if you don't do anything, they work like assignments. (Which is actually a perfectly good parallel to ML, where they work like let expressions, it's just that let expressions don't work like assignments.) > > I'd treat these case branches like 'with' statements. The name binding > from "as" lasts past the end of the block, it's no different from any > other assignment. The only reason exceptions unbind is to avoid the > refloop of an exception having a reference to locals, and the local > name referencing the exception. Everything else can be just like > assignment. Exactly my thought. Binding in Python case statements is different from Haskell case expressions in exactly the same way that assignment statements are different from let expressions, which is pretty much what you should expect. > So basically, what we have here is a cross between a 'with' statement > and an 'if'. I understand why people keep reaching for "with" here, but I really don't like it. Part of the attraction is the "as" clause, which looks like the "as" clause in an ML case expression. But that's missing the point of patterns: what you mostly want to do is decompose the pattern, binding variables to (some of) the components; you don't want to bind a variable to the whole thing. Except that _occasionally_ you _also_ want to bind the whole thing (or bind a subexpression and also some of its own subexpressions), which is what "as" can add. Szymon's insight seems to be that pattern matching can be done, not directly in the way Haskell and ML do it, but instead by having types that know how to transform themselves into simpler structures that Python already knows how to decompose (tuples for tuple unpacking when possible--where the "as" clause is useful--and dicts for updating locals for the less-trivial cases where it's not). That's clever, and it makes good use of an "as" clause, but (especially with the locals-updating dict) it seems so far from what a with statement is intended to do that it seems like an abuse to try to fit his solution into that syntax. Also, "with" just sounds wrong when you read it out loud. Although I'm not sure how much that matters; the same is arguably true with using with to simulate the related but different "if let" statement (as seen in Swift, as Grant sort of proposed in the last email I responded to), but I think I like it there... > I started writing up a demo that mainly used 'with', and > then realized that I had no idea how to have the __enter__ method > stipulate that the body should be skipped... oh, hello PEP 377, that's > where the problem is. There are some hacks for getting around that in CPython 2.6-7 or 3.3+, which you can use for experimenting with it. Or you can always use MacroPy to transform the ASTs before it even gets that far. > This proposal would need some new syntax, but > conceptually, it'd be something like an __enter__ method returning > "hey, this body should be skipped", based on whatever criteria it > likes (isinstance, regex matching, etc). > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abarnert at yahoo.com Wed Apr 8 09:25:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 8 Apr 2015 00:25:15 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> Message-ID: <2F5DC6BB-0D1F-47AD-A864-41D087896F08@yahoo.com> On Apr 7, 2015, at 22:12, Anthony Towns wrote: > >> On 8 April 2015 at 05:25, Szymon Py?alski wrote: >> Any object can be a pattern. If an object has a special method (called >> ``__pattern__`` or ``__match__``) defined then this method will govern >> how this object behaves as pattern. If not - pattern matching works >> the same as ``__eq__`` > >> The ``__pattern__`` method can return ``False`` for no match. For a >> match it can return ``True`` (simplest patterns) a sequence or a mapping > > ?Always returning a sequence or None would seem simpler to deal with, wouldn't it? > I'm imaginging the default implementations would be something like: > > class object: > def __match__(self, item): > if item == self?: > return [item] > else: > return None > > class type: > def __match__(cls, item): > if isinstance(item, cls): > return [item] > else: > return None > > class tuple: > def __match__(self, item): > if isinstance(item, tuple): > if all( si.__match__(ii) for si,ii in zip(self, item) ): > return item > > return None > >> Syntax for pattern matching >> - ------------------------------- >> The syntax could look something like this:: >> >> for object: >> pattern1: statement1 > > Is the two level indent actually a win? It saves re-evaluating the expression, but you could just as easly have: > > object = f(x, y+w, g(z/3 for z in x)) > match object ~~ pattern1 as a,b,c: > statements... > ormatch object ~~ pattern2 as a,d: > statements... > > ?but in that case you're coming pretty close to an 'if' block that's able to set locals: > > object = ... > if object ~~ pattern1 as a,b,c: > ... > elif object ~~ pattern2 as a,d: > ... > > (Using ~~ as a placeholder for the syntax for "matches against") > >> for point: >> (Number, Number) as x, y: >> print_point(x, y) >> {'x': Number, 'y': Number}: >> print_point(x, y) >> re.compile(??'(?P[0-9]+)-(?P[0-9])+'): >> # Or re.compile('([0-9]+)-([0-9])+') as x, y: >> print_point(int(x), int(y)) >> ...: >> raise TypeError('Expected something else') > > My understanding of the way Haskell does name assignment in case matching is that it lets you put placeholders in a constructor. So where you might say > > x = MyX(a=1, b=2) > > to create an object 'x', you could then say something along the lines of: > > x ~~ MyX(a=foo, b=bar) > > to pull the values (1, 2) back out (as foo and bar). Doing exactly that doesn't make sense for python because reversing a constructor doesn't make much sense. This is the big thing that Szymon is trying to add, by adding a __match__ protocol that gives types a way to, in effect, reverse their constructor. That's the same thing I was trying to do in my blog post that I linked earlier. If you don't have that, then what you're building is a strictly weaker version of tuple unpacking (which we already have), together with a less clumsy way to write "try each of these blocks until one of them doesn't raise" (which you could do more generally), not pattern matching. > If you could make that work, you'd get the python version of Haskell pattern matching looking like: > > re_pt = re.compile(?'(?P[0-9]+)-(?P[0-9])+') > case point of: > (Number(x), Number(y)): ... > ('x': Number(x), 'y': Number(y)): ... > re_pt(x=x, y=y): .. > > ?where you're at least mentioning the 'x' and 'y' variables that are getting filled in. > > Okay, probably horrible idea: > > Matching syntax looks like: > > case point is: > (x:=Number, y:=Number): ... > > a:=x is the same as calling assigning_match(locals(), x, "a") run in the local scope. Except, of course, that modifying locals isn't guaranteed to actually affect the locals, and in current CPython it generally doesn't... > a,b:=x is the same as calling assigning_match(locals(), x, ("a", "b")). > > 'case obj is: \n pattern[0]: stmt[0] \n pattern[1]: stmt[1]' does: > > for i in range(2): > try: > pattern[i].__match__(obj) > except NoMatch: > continue > stmt[i] > break > > > __match__() raises NoMatch on failure, or returns the match: > > class object: > def __match__(self, item): > if self == item: > return item > raise NoMatch > > class type: > def __match__(self, item): > if isinstance(item, self): > return item > raise NoMatch > > class tuple: > def __match__(self, item): > if isinstance(item, tuple): > if all( s_i.__match__(i_i) for s_i, i_i in zip(self, item) ): > return item > raise NoMatch > > class dict: > def __match__(self, item): > if isinstance(item, dict): > if item.keys() == self.keys(): > if all(s_v.__match__(item[s_k]) for s_k,s_v in self.items()): > return item > raise NoMatch > > ? class SRE_Pattern: > def __match__(self, item): > m = self.match(it?em) > if m is None: > raise NoMatch > return m.groups() > > ?assigning_match works something like:? > > class assigning_match(object): > def __init__(self, l, matchobj, varname): > assert isinstance(varname, str) or isinstance(varname, tuple) > > self.l = l > self.m = matchobj > self.n = varname > > def __match__(self, item): > r = self.m.__match__(item) > if isinstance(self.n, str): > self.l[self.n] = other > else: > for n, r_i in zip(self.n, r): > self.l[n] = r_i > return r > > Then the example looks like: > > case point is: > (x:=Number, y:=Number): > ... > {"x": x:=Number, "y": y:=Number}: > ... > x,y:=re_pt: > ... This is pretty much my proposal with different syntax. I don't think the := buys you anything, and it doesn't make it obvious how to have a pattern that recursively matches its parts. In my proposal, this would look like: case point: x: Number, y: Number: ... {"x": x: Number, "y": y: Number}: ... re_pt(x, y): ... (My blog post has an "of" for each clause, but I don't think it's necessary. It also doesn't have type annotations or dict matching; since it builds on normal assignment targets instead of reproducing most of the work done there, the obvious way to add those is to add them to assignments, which has other benefits--no more "a = 3 # type: int" in place of "a: int = 3". And meanwhile, I don't think dict matching is all that useful.) > Being able to pull out and name something that's a deep part of an object seems like it could be useful; otherwise it's not adding much over if/elif. Exactly. > Though maybe it'd be interesting to have a "deconstructor" protocol for that instead, ie: > > Foo(a,b,key=c) = foo > > is equivalent to something like: > > x = Foo.__deconstruct__(foo) > a = x[0] > b = x[1] > c = x["key"] That is almost exactly my version of __match__, except for two things. I didn't think through keyword arguments. You've partly solved the problem, but only partly. For example, Foo(a, b, key=c) and Foo(a, sub=b, key=c) may be identical calls, but the deconstructor can only return one of them, and it may not even be the one that was used for construction. I think something like inspect's argspec objects may handle this, but I'd have to work it through. I didn't try to replace everything tuple unpacking does, but to add onto it. We already have the concept of assignment targets and target lists; specifying what happens for each of those, and what happens when something which isn't a target is in a target list, gets you the behavior you want for matching tuples--including matching other iterables, matching first, *rest, and so on--without having to build a complicated tuple.__match__ and then duplicate it in every other sequence type. > I think it might even extend sanely to unpacking subobjects: > > Foo(Bar(a), b) = foo > > becomes: > > x = Foo.__deconstruct__(foo) > Bar(a) = x[0] > b = x[1] > > becomes: > > x = Foo.__deconstruct__(foo) > x2 = Bar.__deconstruct__(x[0]) > a = x2[0] > b = x[1] > > Using "_" to accept and ignore any value would work fine there. If you allowed literals instead of an identifier, you could treat: > > Foo(0, 99, key="x") = foo > > as: > > x = Foo.__deconstruct__(foo) > assert 0 == x[0] > assert 99 == x[1] > assert "x" = x["key"] > > That'd be enough to use it for matching I think: > > case foo is: > Foo(a,b): ... > > would become: > > try: > Foo(a,b) = foo > except: > continue > ... > break Sure. Haskell actually defines pattern-matching let in terms of a translation to case--and pattern-matching function overload definitions too. Once you define either assignment or case, the other two come easy. > But it might be confusing figuring out whether: > > b = 0 > case foo is: > Foo(a,b): print(a) > > should result in value checking: > > if foo ~~ Foo(a,0): print(a) > > or value assignment: > > del b > if foo ~~ Foo(a,b): print(a) This last part is really the only problem. I suggested that it always means "bind to b", not "match the current value or b", with special syntax for when you want the latter. That's effectively what ML-family languages do, and in my experience you usually go a long time before you run into the need to match b's value or read some code that does so, at which point you finally learn the @b or whatever syntax. > Eh, I don't know. Maybe something there is interesting though. > > Cheers, > aj > > -- > Anthony Towns > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Apr 8 11:00:28 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Apr 2015 19:00:28 +1000 Subject: [Python-ideas] Pattern matching In-Reply-To: <29F1D83F-54CF-47B0-8213-DF3E96130ACA@yahoo.com> References: <5523EFBB.7080900@pythonista.net> <985AE3FB-240B-4DCE-8B9A-E3C9421480A9@yahoo.com> <789DA636-7D1F-4E87-A320-9181CD1E5CF0@yahoo.com> <29F1D83F-54CF-47B0-8213-DF3E96130ACA@yahoo.com> Message-ID: On Wed, Apr 8, 2015 at 4:39 PM, Andrew Barnert wrote: >> So basically, what we have here is a cross between a 'with' statement >> and an 'if'. > > I understand why people keep reaching for "with" here, but I really don't like it. > > Part of the attraction is the "as" clause, which looks like the "as" clause in an ML case expression. But that's missing the point of patterns: what you mostly want to do is decompose the pattern, binding variables to (some of) the components; you don't want to bind a variable to the whole thing. Except that _occasionally_ you _also_ want to bind the whole thing (or bind a subexpression and also some of its own subexpressions), which is what "as" can add. > The other similar syntax that I looked at was the from-import. Positing a new keyword "patmatch": regex = re.compile('(?P[0-9]+)-(?P[0-9])+') case line: from regex patmatch x, y: print(x,y) (or possibly combine in the "case" part into a single statement, to remove the two-level indentation and some of the magic) What this means is: 1) Evaluate line and regex (they'd allow arbitrary expressions) 2) Call regex.__patmatch__(line) - I'll call that "ret" 3a) If it returns None, skip to the next patmatch statement. 3b) If it returns a tuple, assign x=ret[0]; y=ret[1] 3c) If it returns a dict, assign x=ret["x"]; y=ret["y"] 3d) If it returns any other object, assign x=ret.x; y=ret.y 4) Discard "ret" itself (ie it never actually gets bound to a name) 5) Execute the block. This would cover all the obvious use-cases. Specifically for cases 3c and 3d, you could also use 'as' syntax to bind to other names: case line: from re.compile('(?P[0-9]+)') patmatch x as y: print(y) This would also deal with ambiguities like namedtuple; you can either use "patmatch x, y" to treat it as a tuple, or "patmatch x as x, y as y" to force attribute access. ChrisA From aj at erisian.com.au Wed Apr 8 14:34:59 2015 From: aj at erisian.com.au (Anthony Towns) Date: Wed, 8 Apr 2015 22:34:59 +1000 Subject: [Python-ideas] Pattern matching In-Reply-To: <2F5DC6BB-0D1F-47AD-A864-41D087896F08@yahoo.com> References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> <2F5DC6BB-0D1F-47AD-A864-41D087896F08@yahoo.com> Message-ID: On 8 April 2015 at 17:25, Andrew Barnert wrote: > On Apr 7, 2015, at 22:12, Anthony Towns wrote: > case point is: > (x:=Number, y:=Number): > ... > {"x": x:=Number, "y": y:=Number}: > ... > x,y:=re_pt: > ... > > This is pretty much my proposal with different syntax. I don't think the > := buys you anything, and it doesn't make it obvious how to have a pattern > that recursively matches its parts. In my proposal, this would look like: > ?Hmm, I thought that was reasonably straightforward. You'd say things like: case rect_coords is: (pos := (left := Number, top := Number), size := (width := Number, height := Number)): ... to pull out rect_coords == (pos, size), pos == (left, top), size == (width, height). Though maybe it'd be interesting to have a "deconstructor" protocol for > that instead, ie: > > Foo(a,b,key=c) = foo > > is equivalent to something like: > > x = Foo.__deconstruct__(foo) > a = x[0] > b = x[1] > c = x["key"] > > > That is almost exactly my version of __match__, except for two things. > > I didn't think through keyword arguments. You've partly solved the > problem, but only partly. For example, Foo(a, b, key=c) and Foo(a, sub=b, > key=c) may be identical calls, but the deconstructor can only return one of > them, and it may not even be the one that was used for construction. I > think something like inspect's argspec objects may handle this, but I'd > have to work it through. > ?I think you could deal with that okay: class type: def __deconstruct__(cls, obj): if not isinstance(obj, cls): raise NotDeconstructable result = obj.__dict__.copy() if callable(cls.__init__): argspec = inspect.getargspec(cls.__init__) for i, argname in enumeraate(argspec[1:]): result[i] = result.get(argname, None) return result So if your have def __init__(a, b, key) and you just store them as attributes, you can access them by name or by the same position as per __init__. I guess you could deconstruct *args and **kwargs too, by pulling out the unused values from result, though you'd probably need to use an ordereddict to keep track of things then. Though that's how recursive unpacking of lists work anyway, isn't it? Match against [head, *tail]? The only other difference is I was just ignoring unspecified bits, maybe it would make more sense to require you to unpack everything in the same way list/tuple unpacking does. Foo(x, y, *args, kw=z, **kwargs) = foo becomes something like: __unpack = Foo.__deconstruct__(foo) __seen = set() __it = iter(__unpack) x = __unpack.pop( next(__it) ) y = __unpack.pop( next(__it) ) z = __unpack.pop( "kw" ) args = [ __unpack.pop(__i) for __i in __unpack ] kwargs = { k: __unpack.pop(k) for k in __unpack.keys() } if __unpack: raise ValueError("too many values to unpack") ?You'd need something fancier than an ordereddict for *args to leave anything for **kwargs to pick up though.? I think that adds up to letting you say: re_point = re.compile('(?P[0-9]+)-(?P[0-9])+') case '1-3' is: re_point(x=xcoord, y=ycoord): print("X coordinate: %s, Y coordinate: %s" % (xcoord, ycoord)) by defining class SRE_Pattern: def __deconstruct__(self, obj): m = self.match(obj) if m is None: raise NotDeconstructable result = m.groupdict() return result which seems plausible. That seems like a prtty friendly syntax for dealing with regexps actually... (I think that's an advantage of having the protocol be Foo.__deconstruct__(obj) rather than obj.__match__() -- I don't think you could handle regexps without having both params) Hmm, as far as literal-checking versus binding goes, if you're using := or "as" to bind subexpressions, as in: case order is: customer, Breakfast(spam, eggs, beans) as breakfast: ... or case order is: customer, breakfast:=Breakfast(spam, eggs, beans) then I /think/ you could use that to distinguish between binding and checking. So: customer = Customer("alice") name = "bob" case customer is: Customer(name): # succeeds, binds name as "alice" case customer is: Customer("bob"): # compares "alice" to "bob", fails # if it had succeeded, wouldn't have bound any variables name = "bob" case customer is: Customer(name as x): # compares "alice" to name ("bob") and fails # but would have bound x as "alice" if it had succeeded That seems like a rule that's reasonably understandable to me? ("expr as name" has definitely grown on me over "name := expr") I guess I'm thinking "literal" matching is just a special case of deconstructing in general, and done by a method like: class object: def __deconstruct__(self, obj): if self == obj: return () else: raise NotDeconstructable That would have the potential side-effect of allowing things like: 0 = n as a valid statement (it'd raise NotDeconstructable if 0 != n). Not sure that's much worse than: [] = n though, which is already allowed. You could have a syntax error if there wasn't anything to bind though. Cheers, aj -- Anthony Towns -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Apr 8 16:07:39 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 8 Apr 2015 07:07:39 -0700 Subject: [Python-ideas] Pattern matching In-Reply-To: References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> <2F5DC6BB-0D1F-47AD-A864-41D087896F08@yahoo.com> Message-ID: <6FD54801-0AF0-4589-9248-8234B058B4F3@yahoo.com> On Apr 8, 2015, at 05:34, Anthony Towns wrote: > >>> On 8 April 2015 at 17:25, Andrew Barnert wrote: > >>> On Apr 7, 2015, at 22:12, Anthony Towns wrote: >>> case point is: >>> (x:=Number, y:=Number): >>> ... >>> {"x": x:=Number, "y": y:=Number}: >>> ... >>> x,y:=re_pt: >>> ... >> >> This is pretty much my proposal with different syntax. I don't think the := buys you anything, and it doesn't make it obvious how to have a pattern that recursively matches its parts. In my proposal, this would look like: > > ?Hmm, I thought that was reasonably straightforward. You'd say things like: > > case rect_coords is: > (pos := (left := Number, top := Number), size := (width := Number, height := Number)): > ... > > to pull out rect_coords == (pos, size), pos == (left, top), size == (width, height). Ah, I see. I was thrown by := looking like assignment in Pascal or some of the mutable ML variants, where it obviously can't be part of a subexpression, but once you get past that, yeah, it's obvious. Sorry. >>> Though maybe it'd be interesting to have a "deconstructor" protocol for that instead, ie: >>> >>> Foo(a,b,key=c) = foo >>> >>> is equivalent to something like: >>> >>> x = Foo.__deconstruct__(foo) >>> a = x[0] >>> b = x[1] >>> c = x["key"] >> >> That is almost exactly my version of __match__, except for two things. >> >> I didn't think through keyword arguments. You've partly solved the problem, but only partly. For example, Foo(a, b, key=c) and Foo(a, sub=b, key=c) may be identical calls, but the deconstructor can only return one of them, and it may not even be the one that was used for construction. I think something like inspect's argspec objects may handle this, but I'd have to work it through. > > ?I think you could deal with that okay: > > class type: > def __deconstruct__(cls, obj): > if not isinstance(obj, cls): > raise NotDeconstructable > > result = obj.__dict__.copy() > if callable(cls.__init__): > argspec = inspect.getargspec(cls.__init__) > for i, argname in enumeraate(argspec[1:]): > result[i] = result.get(argname, None) > > return result > > So if your have def __init__(a, b, key) and you just store them as attributes, you can access them by name or by the same position as per __init__. I guess you could deconstruct *args and **kwargs too, by pulling out the unused values from result, though you'd probably need to use an ordereddict to keep track of things then. But argspec is more powerful and simpler than that--well, not argspec, but fullargspec and especially Signature. Just use inspect.signature and its bind method. That gives you a BoundArguments, and then you can just do boundargs.args[0] to match by position or boundargs.arguments['spam'] to match by keyword; for an argument that could have been passed either way, the value will show up both ways. (And for args that were passed into *args, however they got there, that works too--for example, if you def __init__(self, x, *args) and then do signature(cls.__init__).bind(1, 2, 3), you can see the 3 as args[3] or as arguments['args'][1].) Meanwhile, I'm a bit wary about type implicitly assuming that attributes and __init__ parameters match one-to-one like this. Sure, that's often right, and it will often raise an AttributeError or a TypeError (from Signature.bind) when it's not right--but those cases where it does the wrong thing will be pretty surprising, and hard for a novice to work around. Making classes explicitly opt in to matching seems safer. On the other hand, allowing them to opt in with a dead-simple decorator like @simplematch (or a base class or metaclass or register function or whatever) that just adds a __deconstruct__ method like the one you defined does seem a lot easier than any of the ideas I originally suggested. One last thing here: I like your NotDeconstructable; if we make that a subclass of ValueError, we can change normal tuple unpacking to raise that as well, without breaking any code. > Though that's how recursive unpacking of lists work anyway, isn't it? Match against [head, *tail]? Sure, but that's a pretty bad way to process Python lists (you end up making N copies of average length N/2, plus N stack frames out of a finite limit, and the code ends up more verbose and less readable), so anything that makes that Scheme/ML style look more inviting than iteration is probably not an advantage... > The only other difference is I was just ignoring unspecified bits, maybe it would make more sense to require you to unpack everything in the same way list/tuple unpacking does. > > Foo(x, y, *args, kw=z, **kwargs) = foo > > becomes something like: > > __unpack = Foo.__deconstruct__(foo) > __seen = set() > __it = iter(__unpack) > > x = __unpack.pop( next(__it) ) > y = __unpack.pop( next(__it) ) > z = __unpack.pop( "kw" ) > > args = [ __unpack.pop(__i) for __i in __unpack ] > kwargs = { k: __unpack.pop(k) for k in __unpack.keys() } > > if __unpack: > raise ValueError("too many values to unpack") > > ?You'd need something fancier than an ordereddict for *args to leave anything for **kwargs to pick up though.? > > I think that adds up to letting you say: > > re_point = re.compile('(?P[0-9]+)-(?P[0-9])+') > > case '1-3' is: > re_point(x=xcoord, y=ycoord): > print("X coordinate: %s, Y coordinate: %s" % (xcoord, ycoord)) > > by defining > > class SRE_Pattern: > def __deconstruct__(self, obj): > m = self.match(obj) > if m is None: > raise NotDeconstructable > > result = m.groupdict() > return result > > which seems plausible. That seems like a prtty friendly syntax for dealing with regexps actually... You're starting to turn me around on my dislike of using pattern matching for regexps; that's more readable than existing re code... > (I think that's an advantage of having the protocol be Foo.__deconstruct__(obj) rather than obj.__match__() -- I don't think you could handle regexps without having both params) Yes. I originally thought the extra indirection of letting something pose as the match value's type while actually not being that type was unnecessary, but this case, an re object's type posing as the string's type, seems like a great counterexample. And it doesn't add much complexity, so... I think it's worth it. > Hmm, as far as literal-checking versus binding goes, if you're using := or "as" to bind subexpressions, as in: > > case order is: > customer, Breakfast(spam, eggs, beans) as breakfast: ... > > or > > case order is: > customer, breakfast:=Breakfast(spam, eggs, beans) > > then I /think/ you could use that to distinguish between binding and checking. So: > > customer = Customer("alice") > > name = "bob" > case customer is: > Customer(name): > # succeeds, binds name as "alice" > > case customer is: > Customer("bob"): > # compares "alice" to "bob", fails > # if it had succeeded, wouldn't have bound any variables > > name = "bob" > case customer is: > Customer(name as x): > # compares "alice" to name ("bob") and fails > # but would have bound x as "alice" if it had succeeded > > That seems like a rule that's reasonably understandable to me? I don't know about that. It wouldn't be hard to remember the rule once you learn it, but it may not be that easy to learn, because at first glance it doesn't make sense. It's weird to have "foo" mean "assign to foo" but "foo as bar" in the exact same context mean "use the value of foo, then assign to bar". It's unprecedented in Python (with doesn't assign to the context manager name instead of using it as an expression if you leave off the "as") and I can't think of any other language that does something equivalent. (And I'm pretty sure it would inspire rants about "warts" by a certain person on this list. :) And again, unless use cases for pattern matching in Python turn out to be very different than in ML and friends (which is certainly _possible_, but not something I'd want to just assume--if we don't expect somewhat similar uses, why are we even looking to them for inspiration?), I think the need for matching a variable's value won't be that common, so we shouldn't waste the nicest syntactic form on it... (Although I'm definitely not sold on my throwaway suggestion of "@foo", which has misleading parallels both in Python and in some of the languages the syntax is inspired by...) > ("expr as name" has definitely grown on me over "name := expr") > > > > I guess I'm thinking "literal" matching is just a special case of deconstructing in general, and done by a method like: > > class object: > def __deconstruct__(self, obj): > if self == obj: > return () > else: > raise NotDeconstructable > > That would have the potential side-effect of allowing things like: > > 0 = n > > as a valid statement (it'd raise NotDeconstructable if 0 != n). Not sure that's much worse than: > > [] = n > > though, which is already allowed. You could have a syntax error if there wasn't anything to bind though. Yeah, I tried to come up with a rule that disallowed literal values in an assignment (but not case) target list unless there's some kind of "real pattern" somewhere within (possibly recursively) the target list, but all of the possibilities seem horrible. So I decided the best thing to do was just not mention that "0 = n" is no longer a syntax error, but a legal runtime assertion that hopefully people don't use. Also, while I don't think people should explicitly write "0 = n", I can imagine a code generator or MacroPy macro or whatever that takes generates a pattern based on its arguments that might want to generate something like "0 = n" for the no-args case, and why not allow it? Anyway, I think your additions and changes make a much better proposal than what I originally had, and we're now probably on track to the best Pythonic design for ML-style pattern matching--but I still don't think it's worth adding to Python, or even writing as a PEP for Guido to reject. Have you ever written code where you really missed the feature (except in the first hour back on Python after a week of Haskell hacking)? Or seen any real code that would be substantially improved? If not, this still just seems like a feature for its own sake, or more than one way to do it for no reason. > > Cheers, > aj > > -- > Anthony Towns -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Wed Apr 8 18:24:05 2015 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 8 Apr 2015 12:24:05 -0400 Subject: [Python-ideas] Pattern matching In-Reply-To: <6FD54801-0AF0-4589-9248-8234B058B4F3@yahoo.com> References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> <2F5DC6BB-0D1F-47AD-A864-41D087896F08@yahoo.com> <6FD54801-0AF0-4589-9248-8234B058B4F3@yahoo.com> Message-ID: On Wed, Apr 8, 2015 at 10:07 AM, Andrew Barnert wrote: > On Apr 8, 2015, at 05:34, Anthony Towns wrote: > Anyway, I think your additions and changes make a much better proposal than > what I originally had, and we're now probably on track to the best Pythonic > design for ML-style pattern matching--but I still don't think it's worth > adding to Python, or even writing as a PEP for Guido to reject. Have you > ever written code where you really missed the feature (except in the first > hour back on Python after a week of Haskell hacking)? Or seen any real code > that would be substantially improved? If not, this still just seems like a > feature for its own sake, or more than one way to do it for no reason. All compiling / tree handling code is really annoying without pattern patching. Here is a fragment of code from the ast module: elif isinstance(node, BinOp) and \ isinstance(node.op, (Add, Sub)) and \ isinstance(node.right, Num) and \ isinstance(node.right.n, complex) and \ isinstance(node.left, Num) and \ isinstance(node.left.n, (int, long, float)): Here is what it would look like if we were inside a match statement thingy: # I picked the arg order out of the docs, but it's a bit too "clever" case BinOp(Num(left), (Add | Sub), Num(right)): if isinstance(left, (int, long, float)) and isinstance(right, complex): ... NodeVisitor etc. are hacked up pattern matching-like thingies that mitigates this when they apply. It's harder to demonstrate, but sre_compile is even worse -- it takes in a tree of opcodes and argument vectors, where the argument vector values are totally undocumented etc. (and, IIRC, sometimes they are lists, sometimes they aren't, depending on the opcode). If it used some disjoint union type that unpacked nicely in a match statement, the compiling code would be much more readable, and the internal data structure would be easier to understand. -- Devin From szymon at pythonista.net Wed Apr 8 19:47:30 2015 From: szymon at pythonista.net (=?UTF-8?B?U3p5bW9uIFB5xbxhbHNraQ==?=) Date: Wed, 08 Apr 2015 19:47:30 +0200 Subject: [Python-ideas] Pattern matching In-Reply-To: <9CA5FBB2-878A-4859-BEA1-9D28AFE5C61C@yahoo.com> References: <5523EFBB.7080900@pythonista.net> <55242F21.8010002@pythonista.net> <9CA5FBB2-878A-4859-BEA1-9D28AFE5C61C@yahoo.com> Message-ID: <552569B2.3060609@pythonista.net> On 08.04.2015 00:59, Andrew Barnert wrote: >> "Works the same as __eq__" either doesn't allow you to assign parts while decomposing, or turns out to have a lot of problems. The "works the same as __eq__" referers to simple objects like numbers, strings or booleans. You could for example expect a tuple containing an element that is some metadata with information, how to handle the data. You could then use patterns like (True, Number), (False, Number). >> The mapping is an interesting end-run around the decomposition problem, but it seems like it throws away the type of the object. In other words, Point(x, y) and Vector(x, y) both pattern-match identically to either {'x': x, 'y': y}. In ML or Haskell, distinguishing between the two is one of the key uses of pattern matching. The mapping is only a way to populate the local namespace. If the pattern matches or not is a different notion. >> And another key use is distinguishing Point(0, y), Point(x, 0) and Point(x, y), which it looks like you also can't do this way; the only way to get x bound in the case block (assuming Point, like most classes, isn't iterable and therefore can't be decomposed as a tuple) is to match a dict. I believe that objects that we want to be matchable in this way should be subclasses of a namedtuple.------ > > * Types should match their instances > >> This is an interesting idea. This allows you to do something halfway between taking any value and checking a specific value--and something that would often be useful in Python--which my proposal didn't have. There is a problem with this idea. The isinstance method can accept a tuple. And the tuple means "OR" in this case. In pattern matching however the tuple would match a tuple. This is an ugly inconsistency. Greetings zefciu From ceridwen.mailing.lists at gmail.com Thu Apr 9 15:56:54 2015 From: ceridwen.mailing.lists at gmail.com (Cara) Date: Thu, 9 Apr 2015 09:56:54 -0400 Subject: [Python-ideas] Pattern matching Message-ID: Fair warning: this is long. > I was trying to google if there were some attempts to add pattern > matching feature (as seen in e.g. haskell) to Python. I couldn't however > find anything. Were there any proposals for a feature like this? > > Greetings > zefciu Many. In no particular order: http://pypi.python.org/pypi/pyfpm https://github.com/martinblech/pyfpm http://www.aclevername.com/projects/splarnektity/ https://pypi.python.org/pypi/splarnektity/0.3 http://blog.chadselph.com/adding-functional-style-pattern-matching-to-python.html https://gist.github.com/chadselph/1320421 http://svn.colorstudy.com/home/ianb/recipes/patmatch.py http://monkey.org/~marius/pattern-matching-in-python.html https://github.com/mariusae/match/tree/master https://pypi.python.org/pypi/PEAK-Rules http://peak.telecommunity.com/DevCenter/PEAK-Rules http://home.in.tum.de/~bayerj/patternmatch.py https://github.com/admk/patmat https://github.com/lihaoyi/macropy#pattern-matching These are in addition to the ones Paul gave. There are some more attempts at related areas (unification and other forms of logic programming, multiple dispatch) and discussions without implementations that I didn't list. Before I say anything else, I should disclose my bias: I think pattern matching would be a great addition to Python. Python tries hard to be readable, and pattern matching would make certain kinds of code much more legible than Python currently allows. I'll give two examples, one canonical and one from something I've been working on. The first is red-black tree balancing. Not by accident, rosettacode.org's pattern matching comparison page ( http://rosettacode.org/wiki/Pattern_matching ) uses that as its example for pattern matching in different languages. Most of the implementations fit onto a single page. I won't argue they're marvels of readability, but that's mostly because of too-succinct variable naming. Here's an implementation of the same algorithm in Python: http://scienceblogs.com/goodmath/2008/05/28/the-basic-balanced-search-tree . I'd paste the code, but it's too long to be instructive because the algorithm gets scattered across against five nontrivial functions, each of which has nested conditionals, not counting the implicit nesting from the functions calling one another. The second is performing algebra on the struct module's format strings. I've written some automatic optimization of them with concatenation and scalar multiplication, so that for instance 'b' + 'b' = '2b' rather than 'bb' and '10s' + '10s' = '10s10s'. The value of this is in dealing with format strings that can be combined in various ways to describe the layouts of complicated binary data formats. With some context stripped for brevity, def _check_add(self, other): if not isinstance(other, _StructBase): return NotImplemented if not self.same_endianness(other): return NotImplemented if self.endianness == other.endianness or len(self.endianness) > len(other.endianness): return type(self) else: if isinstance(other, _StructSequence): return type(other) else: return other._STRUCT_SEQUENCES[other.endianness] def __add__(self, other): struct_sequence = self._check_add(other) if isinstance(other, _StructSequence): if other._elements == []: return struct_sequence(self._elements.copy()) else: begin = other._elements[0] rest = other._elements[1:] else: begin = other rest = [] if self._elements == []: rest.insert(0, begin) return struct_sequence(rest) if self._elements[-1].character == begin.character: return struct_sequence(self._elements[:-1] + [self._elements[-1] + begin] + rest) else: return struct_sequence(self._elements + [begin] + rest) The exact details aren't important, my point is this also leads to complicated nested conditionals because to know what actions I need to perform on a given data type, I need checks to make sure the endianness of the two strings matches, multiple type checks, checks for whether one or the other object is empty, and so on, then after verifying what I'm dealing with, decompositions to break a list apart into its first element and remainder and handling the empty-list case. This implementation isn't even complete because it doesn't, for instance, deal with all the possible endianness permutations. There are some simplifications possible, for instance moving all information described by finite sets like endianness into types (so there's a StructSequence, a BigEndianStructSequence, a LittleEndianStructSequence, ...) but this can't handle all the necessary conditionals and doesn't scale well. Those are two small-scale examples. Some broad areas where pattern matching is useful are: 1) Parsing: It's useful both for performing manipulations on grammars like left factoring and operating on the trees that parsers produce. 2) Mathematics: manipulations of abstract mathematical objects like algebraic expressions (not numerical analysis), for instance algebraic simplification rules. 3) Recursive data structures: Python doesn't have algebraic data types or embed recursion into everything the way languages like Haskell do, but there's plenty of recursively-structured data out in the wild like HTML and JSON. 4) Compilers/interpreters: this use is closely related to parsing and recursive data structures because compilers and interpreters are often acting on abstract syntax trees. Often transformations like flattening an AST into bytecode and constant folding can be implemented with pattern matching. I'll stop there because those are the applications I know best, but I'm sure there are others. Pattern matching also subsumes features including multiple dispatch on types that have been proposed, discussed, and implemented many times for Python. I'd argue the existence of more than a dozen versions of pattern matching and their cumulative downloads from PyPi says the desire for the feature isn't zero, but I'm not sure how indicative they are. Whether I've convinced you or not that pattern matching is a worthwhile addition to the language, I'd like it, and if I could, I'd write a library to get it. Past discussions on pattern matching have started and ended with debates about the syntax for adding it as a core language feature. Because the whole point of pattern matching is readability, since you can't do anything with it you couldn't with a sufficiently-complicated nested if-elif-else, arguing about syntax isn't putting the cart before the horse like it is with some features, but there's more to implementing pattern matching than just syntax, and I'd like to expand the discussion to some of those issues. More than that, though, I think pattern matching with imperfect syntax would still be useful, and I'd rather see a mypy-like library now than a language feature that might make it into Python in some distant future. In that spirit, I'll offer some syntax that doesn't involve changing the grammar. To emulate a case expression like Haskell, the only way I can see to do it is abusing 'with' statements because they're the only block-like statement aside from exceptions that allow assignment: with match(expr) as case: with case(pattern) as matches: if matches: target_list = matches suite with case(pattern) as matches: if matches: target_list = matches suite ... Pattern matching fits naturally into for statements and comprehensions: for target_list in pattern(expr): suite (expr for target_list in pattern(expr) if expr) A syntax similar to functools.singledispatch's works for pattern dispatch: @match(pattern) def f(parameter_list): suite @f.case(pattern) def g(parameter_list): suite @f.case(pattern) def h(parameter_list): suite For defining list and dictionary patterns themselves, my best thought is abusing __getitem__, ellipsis, and the slice syntax: Listpattern[1, ..., 'a': int] [1, 'a', 'b', 'c', 2] # matches, name a is bound to 2 [3, 4, 5] # doesn't match because the first element is 1, no binding [1, 2, None] # also doesn't match because None is not an int Dictpattern['a': 1, 'b': 2, 'c': 'a': str] ['a': 1, 'b': 2, 'c': 'foo'] # matches, name a is bound to 'foo' ['a': 1, 'b': 2, 'c': 'bar', 'd': 3] # doesn't match because no ellipsis These are far from the best possible syntax available by changing Python's grammar or syntax, and some of them would benefit from using Macropy to rewrite ASTs. I'm sure they could be improved even within those constraints, and if anyone has ideas, I'm all ears. A lot of work has been done on pattern matching in object-oriented languages with and without algebraic data types. I think Scala's implementation is most mature and well worth looking at for ideas. (Macropy's experimental pattern matching is patterned on Scala's.) "Patterns as Objects in Grace" ( http://homepages.ecs.vuw.ac.nz/~djp/files/DLS2012.pdf ) is a paper on pattern matching in an object-oriented teaching language with structural typing. "Robust Scripting Via Patterns" ( http://hirzels.com/martin/papers/dls12-thorn-patterns.pdf ) is about pattern matching in Thorn, a dynamically-typed language implemented on the JVM. Thorn's pattern matching is integrated into the language at a deep level, and it has some good ideas about using patterns in control structures, some of which I shamelessly stole in writing the above suggestions for pattern matching syntax in Python. I'm just scratching the surface here, much more can be and has been said. I haven't talked at all about the nitty-gritty of implementation, though I can. I think the most productive way to have that discussion on this list would be to look at what smaller features would be most helpful for implementing pattern matching as a library, and likewise for syntax, shades of the recent discussion about macros. Cara From ncoghlan at gmail.com Thu Apr 9 18:07:47 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Apr 2015 12:07:47 -0400 Subject: [Python-ideas] Simpler Customization of Class Creation, next round In-Reply-To: References: Message-ID: On 6 April 2015 at 11:43, Eric Snow wrote: > If the only benefit is the mitigation of metaclass conflicts then > perhaps such conflicts should be addressed directly and we should not > add __init_subclass__. From other threads it sounds like we should be > able to solve metaclass conflicts one way or another. The main intended readability/maintainability benefit is from the perspective of more clearly distinguishing the "customises subclass initialisation" case from the "customises runtime behaviour of subclasses" case. A full custom metaclass doesn't provide any indication of the scope of impact, while __init_subclass__ more clearly indicates that there's no persistent effects on behaviour post-subclass creation. > Another thing I'm unclear on is what will happen if a subclass were to > define its own __init_subclass__? Same thing that happens with any other class method - chaining to the base class implementation would be in the hands of the subclass author via super(). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Fri Apr 10 01:52:49 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 9 Apr 2015 23:52:49 +0000 (UTC) Subject: [Python-ideas] Pattern matching In-Reply-To: References: Message-ID: <843638816.3182228.1428623569480.JavaMail.yahoo@mail.yahoo.com> On Thursday, April 9, 2015 6:57 AM, Cara wrote: > > Fair warning: this is long. > >> I was trying to google if there were some attempts to add pattern >> matching feature (as seen in e.g. haskell) to Python. I couldn't > however >> find anything. Were there any proposals for a feature like this? >> >> Greetings >> zefciu > > Many. In no particular order: > > http://pypi.python.org/pypi/pyfpm > https://github.com/martinblech/pyfpm > > http://www.aclevername.com/projects/splarnektity/ > https://pypi.python.org/pypi/splarnektity/0.3 > > http://blog.chadselph.com/adding-functional-style-pattern-matching-to-python.html > https://gist.github.com/chadselph/1320421 > > http://svn.colorstudy.com/home/ianb/recipes/patmatch.py > > http://monkey.org/~marius/pattern-matching-in-python.html > https://github.com/mariusae/match/tree/master > > https://pypi.python.org/pypi/PEAK-Rules > http://peak.telecommunity.com/DevCenter/PEAK-Rules > > http://home.in.tum.de/~bayerj/patternmatch.py > > https://github.com/admk/patmat > > https://github.com/lihaoyi/macropy#pattern-matching > > These are in addition to the ones Paul gave. There are some more > attempts at related areas (unification and other forms of logic > programming, multiple dispatch) and discussions without implementations > that I didn't list. > > Before I say anything else, I should disclose my bias: I think pattern > matching would be a great addition to Python. Python tries hard to be > readable, and pattern matching would make certain kinds of code much > more legible than Python currently allows. I'll give two examples, one > canonical and one from something I've been working on. > > The first is red-black tree balancing. Not by accident, > rosettacode.org's pattern matching comparison page > ( http://rosettacode.org/wiki/Pattern_matching ) uses that as its > example for pattern matching in different languages. Most of the > implementations fit onto a single page. I won't argue they're marvels > of readability, but that's mostly because of too-succinct variable > naming. Here's an implementation of the same algorithm in Python: > http://scienceblogs.com/goodmath/2008/05/28/the-basic-balanced-search-tree > . I'd paste the code, but it's too long to be instructive because the > algorithm gets scattered across against five nontrivial functions, > each of which has nested conditionals, not counting the implicit > nesting from the functions calling one another. > > The second is performing algebra on the struct module's format strings. > I've written some automatic optimization of them with concatenation and > scalar multiplication, so that for instance 'b' + 'b' = > '2b' rather than > 'bb' and '10s' + '10s' = '10s10s'. The value of > this is in dealing with > format strings that can be combined in various ways to describe the > layouts of complicated binary data formats. With some context stripped > for brevity, > > def _check_add(self, other): > if not isinstance(other, _StructBase): > return NotImplemented > if not self.same_endianness(other): > return NotImplemented > if self.endianness == other.endianness or len(self.endianness) > > len(other.endianness): > return type(self) > else: > if isinstance(other, _StructSequence): > return type(other) > else: > return other._STRUCT_SEQUENCES[other.endianness] > > def __add__(self, other): > struct_sequence = self._check_add(other) > if isinstance(other, _StructSequence): > if other._elements == []: > return struct_sequence(self._elements.copy()) > else: > begin = other._elements[0] > rest = other._elements[1:] > else: > begin = other > rest = [] > if self._elements == []: > rest.insert(0, begin) > return struct_sequence(rest) > if self._elements[-1].character == begin.character: > return struct_sequence(self._elements[:-1] + > [self._elements[-1] + begin] + rest) > else: > return struct_sequence(self._elements + [begin] + rest) This seems to be overcomplicating things by trying to write ML code in Python instead of just writing Python code. For example, the only reason you decompose other._elements into begin and rest is so you can recompose them (a copy of) into the same list you started with. Why bother? (Also, if you're avoiding the limited pattern-matching already available in Python by not writing "begin, *rest = other._elements", that doesn't make a great case for adding more.) def __add__(self, other): struct_sequence = self._check_add(other) if isinstance(other, _StructSequence): elements = other._elements else: elements = [other] if (self._elements and elements and self.elements[-1].character == elements[0].character): return struct_sequence(self._elements[:-1] + [self._elements[-1] + elements[0]] + elements[1:]) return struct_sequence(self._elements + elements) If this is all you're doing, your argument is equivalent to showing some inherently iterative code written instead in tail-recursive form to argue that Python should do tail call elimination. You're asking to make it easier to provide another, less Pythonic way to do something that can already be easily done Pythonically. > The exact details aren't important, my point is this also leads to > complicated nested conditionals because to know what actions I need to > perform on a given data type, I need checks to make sure the endianness > of the two strings matches, multiple type checks, checks for whether one > or the other object is empty, and so on, then after verifying what I'm > dealing with, decompositions to break a list apart into its first > element and remainder and handling the empty-list case. But everything from "decompositions" on is only necessary because you're thinking in terms of decomposition instead of list operations. > This > implementation isn't even complete because it doesn't, for instance, > deal with all the possible endianness permutations. There are some > simplifications possible, for instance moving all information described > by finite sets like endianness into types (so there's a StructSequence, > a BigEndianStructSequence, a LittleEndianStructSequence, ...) but this > can't handle all the necessary conditionals and doesn't scale well. > > Those are two small-scale examples. Some broad areas where pattern > matching is useful are: > > 1) Parsing: It's useful both for performing manipulations on grammars > like left factoring and operating on the trees that parsers produce. Parsing, and related things (compiling, schema validation, etc.) looks like a good case, but I'm not sure it is. For simple languages, an expression-tree library like PyParsing or Boost.Spirit seems to be more than sufficient, and often be even nicer than explicit pattern matching. For more complex languages (and sometimes even for simple ones), you can use or write a framework that lets you write a fully-declarative syntax and then executes it or generates executable code from it. For example, if you want to validate and parse an XML language, doing it by hand in ML would be nicer than in Python, but writing it in XML Schema and using a schema-validating parser is much nicer than either. Parsing the Python grammar by hand would be nicer in ML than in C or Python, but writing it as a declarative GRAMMAR file is nicer than either. And so on. You could argue that the work to write that framework is wasted if it's only going to be used once, but how many complex languages really need to be parsed only once?Even CPython's parser, which isn't used for anything but parsing Python's GRAMMAR, is effectively reused every time Python's syntax changes, and every time someone wants to hack on it to play with new syntax. The one really good case for pattern matching seems to be when you pretty much _have_ to write a custom parser--maybe because the language is full of special-case rules and tangled context, like C++. At least the GCC and LLVM/Clang groups gave up trying to write C++ as a declarative specification and instead wrote custom recursive-descent parsers in C, which I suspect could be a lot nicer with pattern matching. But has anyone written an equivalent C++ parser in Haskell, ML, etc. to compare, or is that just a suspicion? > 2) Mathematics: manipulations of abstract mathematical objects like > algebraic expressions (not numerical analysis), for instance algebraic > simplification rules. Are you talking about parsing the expressions into a tree? If so, we're still on parsing. > 3) Recursive data structures: Python doesn't have algebraic data types > or embed recursion into everything the way languages like Haskell do, > but there's plenty of recursively-structured data out in the wild like > HTML and JSON. The parsing of this data is covered above. But what do you do with it once it's parsed? If you're transforming it, that again seems to be a case where writing the transformer declaratively in, say, XSSL wins. And if you're just extracting data from it, usually you're better off with the simple data['machines'][0]['name'] syntax (in an except KeyError block), and when that's not good enough, you're usually better off with a declarative search language like XPath. So this seems to be the same as with parsing. > 4) Compilers/interpreters: this use is closely related to parsing and > recursive data structures because compilers and interpreters are often > acting on abstract syntax trees. Often transformations like flattening > an AST into bytecode and constant folding can be implemented with > pattern matching. > > I'll stop there because those are the applications I know best, but I'm > sure there are others. Pattern matching also subsumes features > including multiple dispatch on types that have been proposed, discussed, > and implemented many times for Python. I'd argue the existence of more > than a dozen versions of pattern matching and their cumulative downloads > from PyPi says the desire for the feature isn't zero, but I'm not sure > how indicative they are. > > Whether I've convinced you or not that pattern matching is a worthwhile > addition to the language, I'd like it, and if I could, I'd write a > library to get it. That raises the question: what's wrong with the more than a dozen libraries already written? > Past discussions on pattern matching have started > and ended with debates about the syntax for adding it as a core language > feature. Because the whole point of pattern matching is readability, > since you can't do anything with it you couldn't with a > sufficiently-complicated nested if-elif-else, arguing about syntax isn't > putting the cart before the horse like it is with some features, but > there's more to implementing pattern matching than just syntax, and I'd > like to expand the discussion to some of those issues. More than that, > though, I think pattern matching with imperfect syntax would still be > useful, and I'd rather see a mypy-like library now than a language > feature that might make it into Python in some distant future. In that > spirit, I'll offer some syntax that doesn't involve changing the > grammar. To emulate a case expression like Haskell, the only way I can > see to do it is abusing 'with' statements because they're the only > block-like statement aside from exceptions that allow assignment: > > with match(expr) as case: > with case(pattern) as matches: > if matches: > target_list = matches > suite > with case(pattern) as matches: > if matches: > target_list = matches > suite > ... First, this seems like it will execute _every_ branch that matches, instead of just the first one. Surely you don't want that, do you? You can wrap the whole thing in a "while True:" and have each suite (and the "else" case) end with "break", or wrap it in a try/except and have each suite raise a custom exception, but you need some way to get out. Also, why not just put the target_list in the as clause (inside parens, to fit it in a target, which is what an as clause actually takes)? Of course that means you get an exception rather than an else when it doesn't match, but that's fine. (It would be nice if it were a subclass of ValueError, in case the suite raises some ValueError of its own, however...) > Pattern matching fits naturally into for statements and comprehensions: > > for target_list in pattern(expr): > suite What is this going to do? From your above example, if looks like match(expr)(pattern) returns either an iterable that can be assigned to a target_list or None. What does pattern(expr) return? Return an iterable of one target_list or an empty iterable? Also, the very idea of having an object which can be called on an expression or can have a (processed) expression called on it with similar but not identical results seems very confusing. > (expr for target_list in pattern(expr) if expr) > > A syntax similar to functools.singledispatch's works for pattern > dispatch: > > @match(pattern) > def f(parameter_list): > suite > > @f.case(pattern) > def g(parameter_list): > suite > > @f.case(pattern) > def h(parameter_list): > suite > > For defining list and dictionary patterns themselves, my best thought is > abusing __getitem__, ellipsis, and the slice syntax: > > Listpattern[1, ..., 'a': int] > > [1, 'a', 'b', 'c', 2] # matches, name a is bound to 2 > [3, 4, 5] # doesn't match because the first element is 1, no binding > [1, 2, None] # also doesn't match because None is not an int > > Dictpattern['a': 1, 'b': 2, 'c': 'a': str] > > ['a': 1, 'b': 2, 'c': 'foo'] # matches, name a > is bound to 'foo' > ['a': 1, 'b': 2, 'c': 'bar', 'd': 3] # > doesn't match because no ellipsis The useful part of pattern matching--and the hard part, too--is decomposing structures. A proposal that ignores that and only decomposes lists and dicts misses the point. Yes, it's true that Python doesn't have ADTs. So that means one of two things: (1) types that want to be matched will have to do a bit of work to expose their structure and/or to decompose themselves, or (2) pattern matching isn't useful in Python. The examples you give from other languages show that the first is at least plausible. And both Szymon's proposal and mine and various other ideas suggested on this thread show attempts at a Pythonic design. But just ignoring structure decomposition, as you're doing, is not a way toward anything useful. > These are far from the best possible syntax available by changing > Python's grammar or syntax, and some of them would benefit from using > Macropy to rewrite ASTs. I'm sure they could be improved even within > those constraints, and if anyone has ideas, I'm all ears. If you're willing to use MacroPy, that brings up the question: what's wrong with the pattern matching that it comes with as an example, and why don't you want to build on it instead of starting over? > A lot of work has been done on pattern matching in object-oriented > languages with and without algebraic data types. I think Scala's > implementation is most mature and well worth looking at for ideas. > (Macropy's experimental pattern matching is patterned on Scala's.) > "Patterns as Objects in > Grace" ( http://homepages.ecs.vuw.ac.nz/~djp/files/DLS2012.pdf ) is a > paper on pattern matching in an object-oriented teaching language with > structural typing. "Robust Scripting Via > Patterns" ( http://hirzels.com/martin/papers/dls12-thorn-patterns.pdf ) > is about pattern matching in Thorn, a dynamically-typed language > implemented on the JVM. Thorn's pattern matching is integrated into the > language at a deep level, and it has some good ideas about using > patterns in control structures, some of which I shamelessly stole in > writing the above suggestions for pattern matching syntax in Python. > I'm just scratching the surface here, much more can be and has been > said. > > I haven't talked at all about the nitty-gritty of implementation, though > I can. I think the most productive way to have that discussion on this > list would be to look at what smaller features would be most helpful for > implementing pattern matching as a library, and likewise for syntax, > shades of the recent discussion about macros. This last part, I think, is very useful: After building the best pattern-matching library you can with Python today, figure out what small changes to the language would allow you to either make it better or simplify its implementation, then propose those small changes instead of proposing pattern matching itself. For example, if you have to use nonportable hacks (or a MacroPy macro) so your context manager's __enter__ method can skip the statement body, that would be a good argument for reopening PEP 377 (because you have a new use case that wasn't known at the time). But trying to guess in advance what features might be helpful seems harder and less likely to pay off. That being said, I suspect it's not going to pay off, because I think the with statement is a false lead in the first place. The semantics look promising, but they're not actually what you want to build on (unless you want to completely transform with statements into something that isn't actually a with statement, using MacroPy AST transformations or otherwise). I'd love to be proven wrong, of course. Meanwhile transforming a pattern-match into something that can be assigned to a target_list instead of just assigning in the match does work in simple cases, but I think Szymon's proposal shows why it's not sufficient, and I think any real examples will show that it's redundant. Even without the language support in my proposal, explicit quoting of names like in Grant Jenks' PyPatt library shows that you can get most of the way there. (The "as" clause is useful for the case when you want to bind the whole structure at the same time you decompose it and potentially bind its parts, but that's pretty rare--which is another reason I think "with" is misleading.) > > Cara > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From travis.a.everett at gmail.com Fri Apr 10 21:54:32 2015 From: travis.a.everett at gmail.com (Travis Everett) Date: Fri, 10 Apr 2015 14:54:32 -0500 Subject: [Python-ideas] notify exception during raise (once magic properties are set) Message-ID: Hi all, I have a specialized use case for exceptions which need to analyze their __cause__ property before they're fully initialized. The property isn't set until after init, and (forgive my ignorance of Python's source!) appears to be done at the C level in a way that it can't be observed with a property setter or __setattr__. I currently create a method to do this post-init work as a workaround, and push the burden of making sure it gets called on the code using the exceptions--but this makes the exceptions idiomatic and error-prone. A magic method called once all of the special exception properties are already set would suffice. The exception could define something like __raised__(self) to react to the new values of those properties. A more comprehensive approach might be a __raise__(self, cause, context, traceback...) method which has full control over setting the properties, but I assume there are good reasons why the usual channels (setters and __setattr__) are cut out of the loop. I was encouraged to bring discussion here (to see if there's any traction) after opening an enhancement request ( http://bugs.python.org/issue23902). Cheers, Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Apr 10 22:21:45 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Apr 2015 06:21:45 +1000 Subject: [Python-ideas] notify exception during raise (once magic properties are set) In-Reply-To: References: Message-ID: On Sat, Apr 11, 2015 at 5:54 AM, Travis Everett wrote: > I have a specialized use case for exceptions which need to analyze their > __cause__ property before they're fully initialized. The property isn't set > until after init, and (forgive my ignorance of Python's source!) appears to > be done at the C level in a way that it can't be observed with a property > setter or __setattr__. I currently create a method to do this post-init work > as a workaround, and push the burden of making sure it gets called on the > code using the exceptions--but this makes the exceptions idiomatic and > error-prone. On principle I'd be in favour of making the setting of __cause__ go through the normal mechanisms, such that @property etc would work; but just out of curiosity, what would happen if your code raised an exception during the raising of another exception? Because that's when __cause__ will get set - the new exception object is fully constructed, and now it goes into exception handling to start bubbling up. Could be a bit weird! But if that's handlable (maybe it just chains yet another exception in), then that's the route I'd be inclined towards. There are quite a few things that happen somewhat weirdly in Python, largely (I think) because of specific CPython optimizations or implementation details. Anything that makes the language more flexible may well have a hefty performance cost, but if it doesn't, then it would be a good thing IMO. ChrisA From abarnert at yahoo.com Fri Apr 10 23:39:33 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 10 Apr 2015 14:39:33 -0700 Subject: [Python-ideas] notify exception during raise (once magic properties are set) In-Reply-To: References: Message-ID: <21CDB9C8-55F9-47AF-A564-C2FA7584DC6D@yahoo.com> On Apr 10, 2015, at 12:54, Travis Everett wrote: > > Hi all, > > I have a specialized use case for exceptions which need to analyze their __cause__ property before they're fully initialized. The property isn't set until after init, and (forgive my ignorance of Python's source!) appears to be done at the C level in a way that it can't be observed with a property setter or __setattr__. I currently create a method to do this post-init work as a workaround, and push the burden of making sure it gets called on the code using the exceptions--but this makes the exceptions idiomatic and error-prone. > > A magic method called once all of the special exception properties are already set would suffice. The exception could define something like __raised__(self) to react to the new values of those properties. > > A more comprehensive approach might be a __raise__(self, cause, context, traceback...) method which has full control over setting the properties, but I assume there are good reasons why the usual channels (setters and __setattr__) are cut out of the loop. I'd assume the only reasons are to make the code a bit simpler and a bit more efficient. Adding some new __raise__ or __raised__ special-purpose protocol doesn't seem like it would be any simpler or more efficient than just changing the existing APIs to access the attributes via SetAttr, which would give you what you want without being a special case. Also notice that there's an open PEP (https://www.python.org/dev/peps/pep-0490/) to make the C API for raising exceptions automatically set the context instead of forcing it to be done manually (with a private API function), as a few builtin and stdlib functions do. So it seems safer to change things in a way that would work equally well with the current APIs, with that change, and with all reasonable alternatives to that change--and again, using SetAttr seems like the simplest way to make sure of that. But either way, it's a relatively local and simple change in Objects/exceptions.c. I suspect the big thing you'd need to get this accepted is to show that the performance cost is negligible. Do you know how to run appropriate benchmarks and present the results if someone else writes the patch? > I was encouraged to bring discussion here (to see if there's any traction) after opening an enhancement request (http://bugs.python.org/issue23902). > > Cheers, > Travis > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Apr 11 00:07:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 10 Apr 2015 15:07:15 -0700 Subject: [Python-ideas] notify exception during raise (once magic properties are set) In-Reply-To: <21CDB9C8-55F9-47AF-A564-C2FA7584DC6D@yahoo.com> References: <21CDB9C8-55F9-47AF-A564-C2FA7584DC6D@yahoo.com> Message-ID: On Apr 10, 2015, at 14:39, Andrew Barnert wrote: > >> On Apr 10, 2015, at 12:54, Travis Everett wrote: >> >> Hi all, >> >> I have a specialized use case for exceptions which need to analyze their __cause__ property before they're fully initialized. The property isn't set until after init, and (forgive my ignorance of Python's source!) appears to be done at the C level in a way that it can't be observed with a property setter or __setattr__. I currently create a method to do this post-init work as a workaround, and push the burden of making sure it gets called on the code using the exceptions--but this makes the exceptions idiomatic and error-prone. >> >> A magic method called once all of the special exception properties are already set would suffice. The exception could define something like __raised__(self) to react to the new values of those properties. >> >> A more comprehensive approach might be a __raise__(self, cause, context, traceback...) method which has full control over setting the properties, but I assume there are good reasons why the usual channels (setters and __setattr__) are cut out of the loop. > > I'd assume the only reasons are to make the code a bit simpler and a bit more efficient. Actually, on second though, it's not _quite_ that simple. First, settings args, __traceback__ etc. are no longer hookable; it seems like you'd want those to be as well, for consistency? So that's a few more functions to change. Second, the API functions for setting cause and context (and the private API function for automatically setting context) can't possibly fail anywhere, so they have no return value. Changing them to go through SetAttr means they now can (your __setattr__ could raise anything it wants--or even creating the 1 object to store in __suppress_context__ could fail). Should we just swallow and ignore any such exceptions when called from the C error API? (C code that uses the exception type API or the object API will of course continue to get exceptions, as will Python code, it's just the error machinery that would ignore errors from munging exceptions to be raised.) > Adding some new __raise__ or __raised__ special-purpose protocol doesn't seem like it would be any simpler or more efficient than just changing the existing APIs to access the attributes via SetAttr, which would give you what you want without being a special case. > > Also notice that there's an open PEP (https://www.python.org/dev/peps/pep-0490/) to make the C API for raising exceptions automatically set the context instead of forcing it to be done manually (with a private API function), as a few builtin and stdlib functions do. So it seems safer to change things in a way that would work equally well with the current APIs, with that change, and with all reasonable alternatives to that change--and again, using SetAttr seems like the simplest way to make sure of that. > > But either way, it's a relatively local and simple change in Objects/exceptions.c. > > I suspect the big thing you'd need to get this accepted is to show that the performance cost is negligible. Do you know how to run appropriate benchmarks and present the results if someone else writes the patch? > >> I was encouraged to bring discussion here (to see if there's any traction) after opening an enhancement request (http://bugs.python.org/issue23902). > > >> Cheers, >> Travis >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 11 13:29:32 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Apr 2015 21:29:32 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses Message-ID: A bare "except:" clause catches any exception. As of Python 3.0, this can be spelled more explicitly as "except BaseException:", and AFAIK the only difference between the two is that someone might rebind BaseException. String exceptions were abolished during the 2.x line, without causing major upheaval. The deprecation and ultimate abolition of the bare except syntax would have less consequence than that, as it would merely require a mechanical transformation to "except BaseException:", without needing any corresponding changes at raise site. Pros: * Remove the attractive nuisance of "hmm, my code's throwing an error that I don't recognize, I'll use try/except" and just catching everything. * Eliminates the edge case wherein "except:" and "except ():" are almost diametrically opposite in meaning. Not often a concern, but we've just had a lengthy thread on python-list discussing this. It's generally agreed that an empty tuple has to be interpreted as catching nothing, but the false parallel with an empty clause is problematic. Cons: * May make it harder to write Py2/Py3-compatible code. In Py2, "except:" will catch old-style-class exceptions. * Any removal of any feature can cause examples and published code to break. Undecided: * Forcing everyone to look at the code that uses "except:" means extra work to upgrade Python, but it might mean they notice a problem somewhere. Maybe it should be "except Exception", or maybe something even more specific. Is this a pro or a con? Thoughts? ChrisA From storchaka at gmail.com Sat Apr 11 14:10:03 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 11 Apr 2015 15:10:03 +0300 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: Message-ID: On 11.04.15 14:29, Chris Angelico wrote: > A bare "except:" clause catches any exception. As of Python 3.0, this > can be spelled more explicitly as "except BaseException:", and AFAIK > the only difference between the two is that someone might rebind > BaseException. > > String exceptions were abolished during the 2.x line, without causing > major upheaval. The deprecation and ultimate abolition of the bare > except syntax would have less consequence than that, as it would > merely require a mechanical transformation to "except BaseException:", > without needing any corresponding changes at raise site. > > Pros: > * Remove the attractive nuisance of "hmm, my code's throwing an error > that I don't recognize, I'll use try/except" and just catching > everything. This doesn't different from the use of "except BaseException:". > * Eliminates the edge case wherein "except:" and "except ():" are > almost diametrically opposite in meaning. Not often a concern, but > we've just had a lengthy thread on python-list discussing this. It's > generally agreed that an empty tuple has to be interpreted as catching > nothing, but the false parallel with an empty clause is problematic. Is this a problem? > Cons: > * May make it harder to write Py2/Py3-compatible code. In Py2, > "except:" will catch old-style-class exceptions. I think this is enough to not abolish bare except clauses at least while 2.7 in the use. From rosuav at gmail.com Sat Apr 11 14:28:17 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Apr 2015 22:28:17 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: Message-ID: On Sat, Apr 11, 2015 at 10:10 PM, Serhiy Storchaka wrote: >> >> Pros: >> * Remove the attractive nuisance of "hmm, my code's throwing an error >> that I don't recognize, I'll use try/except" and just catching >> everything. > > > This doesn't different from the use of "except BaseException:". Apart from the fact that "except:" is short and convenient. If someone were to quickly scribble something down, it's more likely to be Exception than BaseException, which at very least would prevent the catching of KeyboardInterrupt and friends; and since you're forced to put _something_ in, there's a better chance that you'll go and put the right thing in. >> * Eliminates the edge case wherein "except:" and "except ():" are >> almost diametrically opposite in meaning. Not often a concern, but >> we've just had a lengthy thread on python-list discussing this. It's >> generally agreed that an empty tuple has to be interpreted as catching >> nothing, but the false parallel with an empty clause is problematic. > > > Is this a problem? Hardly a huge one, but it's certainly not the other way around. >> Cons: >> * May make it harder to write Py2/Py3-compatible code. In Py2, >> "except:" will catch old-style-class exceptions. > > > I think this is enough to not abolish bare except clauses at least while 2.7 > in the use. Are old-style classes often thrown? And is there any advantage to not just adding "(Exception)" to the classes' definitions? Currently, the bare except clause is even weirder in 2.7, because there are now three distinct levels of "catch everything": # what most people should think of as "everything" except Exception as e: # what the exception hierarchy sees as "everything" except BaseException as e: # absolutely everything except: Plus, the bare except clause doesn't allow capturing. One good use of "catch everything" is the generic logger at a significant boundary (maybe a web server that runs application code; if the app chokes, the server should catch that, log it, and return a 500), and this is nice and tidy with "except BaseException as e", but with "except:", you have to go digging. It's in many ways a wart; what benefit is it, really? ChrisA From abarnert at yahoo.com Sat Apr 11 14:25:44 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 11 Apr 2015 05:25:44 -0700 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: Message-ID: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> On Apr 11, 2015, at 04:29, Chris Angelico wrote: > > A bare "except:" clause catches any exception. As of Python 3.0, this > can be spelled more explicitly as "except BaseException:", and AFAIK > the only difference between the two is that someone might rebind > BaseException. > > String exceptions were abolished during the 2.x line, without causing > major upheaval. I couldn't remember when this happened, so I look at the docs... But 2.7 (https://docs.python.org/2.7/reference/executionmodel.html#exceptions) still say "Exceptions can also be identified by strings, in which case the except clause is selected by object identity." Do the docs need to be fixed? Meanwhile, in the C API (https://docs.python.org/2.7/c-api/exceptions.html#string-exceptions), it says "Changed in version 2.6: All exceptions to be raised or caught must be derived from BaseException." I think that one is correct, but until I sit down at a bigger screen where I can browse the source or run an interpreter, I'm not sure. > The deprecation and ultimate abolition of the bare > except syntax would have less consequence than that, as it would > merely require a mechanical transformation to "except BaseException:", > without needing any corresponding changes at raise site. > > Pros: > * Remove the attractive nuisance of "hmm, my code's throwing an error > that I don't recognize, I'll use try/except" and just catching > everything. But if you just teach people "spell that as except BaseException:" (or, more likely, "... as except Exception:"), it's still the exact same attractive nuisance. And, because "except:" is almost always a red flag in novice code, but "except Exception:" is occasionally reasonable, you're making it a tiny bit harder to spot the mistake when helping novices, without actually fixing it. (Of course requiring an "as" clause would solve all of this, but nobody wants that...) > * Eliminates the edge case wherein "except:" and "except ():" are > almost diametrically opposite in meaning. Not often a concern, but > we've just had a lengthy thread on python-list discussing this. It's > generally agreed that an empty tuple has to be interpreted as catching > nothing, but the false parallel with an empty clause is problematic. > > Cons: > * May make it harder to write Py2/Py3-compatible code. In Py2, > "except:" will catch old-style-class exceptions. If the C API docs are right and exceptions must be derived from BaseException even in 2.6, they can't be old-style class instances in the first place. (And if anyone's trying to write 2.5/3.x code, they still have to deal with string exceptions, not to mention that they can't get the value of any exceptions they handle, so they've got much bigger problems.) Even if that's not right, what problems do old-style classes cause that new-style-but-not-BaseException-derived classes don't? > * Any removal of any feature can cause examples and published code to break. > > Undecided: > * Forcing everyone to look at the code that uses "except:" means extra > work to upgrade Python, but it might mean they notice a problem > somewhere. Maybe it should be "except Exception", or maybe something > even more specific. Is this a pro or a con? > > Thoughts? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 11 14:39:04 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Apr 2015 22:39:04 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> Message-ID: On Sat, Apr 11, 2015 at 10:25 PM, Andrew Barnert wrote: > On Apr 11, 2015, at 04:29, Chris Angelico wrote: > > A bare "except:" clause catches any exception. As of Python 3.0, this > can be spelled more explicitly as "except BaseException:", and AFAIK > the only difference between the two is that someone might rebind > BaseException. > > String exceptions were abolished during the 2.x line, without causing > major upheaval. > > > I couldn't remember when this happened, so I look at the docs... But 2.7 > (https://docs.python.org/2.7/reference/executionmodel.html#exceptions) still > say "Exceptions can also be identified by strings, in which case the except > clause is selected by object identity." Do the docs need to be fixed? 1) Yes, sounds like it. 2) By identity? Not by equality?? 3) I have absolutely no idea about string exceptions - I do my best to avoid them. :) > Meanwhile, in the C API > (https://docs.python.org/2.7/c-api/exceptions.html#string-exceptions), it > says "Changed in version 2.6: All exceptions to be raised or caught must be > derived from BaseException." I think that one is correct, but until I sit > down at a bigger screen where I can browse the source or run an interpreter, > I'm not sure. Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> raise 1 Traceback (most recent call last): File "", line 1, in TypeError: exceptions must be old-style classes or derived from BaseException, not int >>> class OldException: pass ... >>> raise OldException Traceback (most recent call last): File "", line 1, in __main__.OldException: <__main__.OldException instance at 0x7f0e6c89e1b8> So that one probably needs a docs change too, although it's not as serious a problem. > Pros: > * Remove the attractive nuisance of "hmm, my code's throwing an error > that I don't recognize, I'll use try/except" and just catching > everything. > > > But if you just teach people "spell that as except BaseException:" (or, more > likely, "... as except Exception:"), it's still the exact same attractive > nuisance. And, because "except:" is almost always a red flag in novice code, > but "except Exception:" is occasionally reasonable, you're making it a tiny > bit harder to spot the mistake when helping novices, without actually fixing > it. > > (Of course requiring an "as" clause would solve all of this, but nobody > wants that...) So you don't teach people to spell it as "except Exception". You teach them to look up the actual exception type they want to catch. That already happens, but I've seen quite a bit of code from my own students where the shorthand of "except:" is so tempting that it gets in. If that came straight back with SyntaxError, they'd be forced to put _something_ in, and would be more likely to put in a useful exception type. > Cons: > * May make it harder to write Py2/Py3-compatible code. In Py2, > "except:" will catch old-style-class exceptions. > > > If the C API docs are right and exceptions must be derived from > BaseException even in 2.6, they can't be old-style class instances in the > first place. (And if anyone's trying to write 2.5/3.x code, they still have > to deal with string exceptions, not to mention that they can't get the value > of any exceptions they handle, so they've got much bigger problems.) Yeah, the wide-span compatibility issue is always going to be there. But given that the docs aren't quite right here, this is an issue with 2.7/3.x code. > Even if that's not right, what problems do old-style classes cause that > new-style-but-not-BaseException-derived classes don't? I don't know. I've never consciously raised or caught an old-style class. If I unwittingly did, there'd be no problem to any of my code if it suddenly started inheriting from Exception or StandardError. ChrisA From tritium-list at sdamon.com Sat Apr 11 15:11:56 2015 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 11 Apr 2015 09:11:56 -0400 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> Message-ID: <55291D9C.2020305@sdamon.com> On 4/11/2015 08:25, Andrew Barnert wrote: > But if you just teach people "spell that as except BaseException:" > (or, more likely, "... as except Exception:"), it's still the exact > same attractive nuisance. And, because "except:" is almost always a > red flag in novice code, but "except Exception:" is occasionally > reasonable, you're making it a tiny bit harder to spot the mistake > when helping novices, without actually fixing it. I think it is kind of a false argument to say 'this should not be considered because people will just break it another way'. Bare excepts were a mistake. I for one would support removing them in 2.7.10, but that's not on the table. As long as the 'new valid' syntax is also valid 2.7-3.5, it should be considered. As I understand the proposal, `except ASubclassOfException:` would remain valid - an Exception type or object being required for an except clause is not a massive break in 2.7 (its already required there if you are not using bare excepts). The burden of supporting 2.7 through a hypothetical 3.6 without being able to use bare exceptions is minimal to none. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Apr 11 15:16:24 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 11 Apr 2015 06:16:24 -0700 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> Message-ID: On Apr 11, 2015, at 05:39, Chris Angelico wrote: > >> On Sat, Apr 11, 2015 at 10:25 PM, Andrew Barnert wrote: >> On Apr 11, 2015, at 04:29, Chris Angelico wrote: >> >> A bare "except:" clause catches any exception. As of Python 3.0, this >> can be spelled more explicitly as "except BaseException:", and AFAIK >> the only difference between the two is that someone might rebind >> BaseException. >> >> String exceptions were abolished during the 2.x line, without causing >> major upheaval. >> >> >> I couldn't remember when this happened, so I look at the docs... But 2.7 >> (https://docs.python.org/2.7/reference/executionmodel.html#exceptions) still >> say "Exceptions can also be identified by strings, in which case the except >> clause is selected by object identity." Do the docs need to be fixed? > > 1) Yes, sounds like it. That worries me a bit. If the docs say strings can be raised in 2.7, even if CPython doesn't allow it, I think you pretty much have to survey every implementation and make sure none of them allow it either before you can feel safe saying it's _just_ a doc problem... > 2) By identity? Not by equality?? Yeah, I vaguely remember the rule in the old days (as in 2.3-2.5-ish) was identity-or-isinstance. Of course if you can't actually raise strings in 2.6 and 2.7 (from either Python or C), how they'd be identified if they _were_ raised is kind of irrelevant... > 3) I have absolutely no idea about string exceptions - I do my best to > avoid them. Well, yeah, but if they really do still exist in 2.7 or even 2.6, I think that would count as a strike against removing bare except. > :) > >> Meanwhile, in the C API >> (https://docs.python.org/2.7/c-api/exceptions.html#string-exceptions), it >> says "Changed in version 2.6: All exceptions to be raised or caught must be >> derived from BaseException." I think that one is correct, but until I sit >> down at a bigger screen where I can browse the source or run an interpreter, >> I'm not sure. > > Python 2.7.3 (default, Mar 13 2014, 11:03:55) > [GCC 4.7.2] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> raise 1 > Traceback (most recent call last): > File "", line 1, in > TypeError: exceptions must be old-style classes or derived from > BaseException, not int >>>> class OldException: pass > ... >>>> raise OldException > Traceback (most recent call last): > File "", line 1, in > __main__.OldException: <__main__.OldException instance at 0x7f0e6c89e1b8> > > So that one probably needs a docs change too, although it's not as > serious a problem. Maybe it means you can't raise them through the public C API, but you can still raise them through Python (or private C functions, presumably)? (But it also says "caught"; does that mean C code can't use the PyErr_GivenExceptionMatches function to catch your OldException? That would be weird... But not entirely relevant here, so let's not worry about it.) >> Pros: >> * Remove the attractive nuisance of "hmm, my code's throwing an error >> that I don't recognize, I'll use try/except" and just catching >> everything. >> >> >> But if you just teach people "spell that as except BaseException:" (or, more >> likely, "... as except Exception:"), it's still the exact same attractive >> nuisance. And, because "except:" is almost always a red flag in novice code, >> but "except Exception:" is occasionally reasonable, you're making it a tiny >> bit harder to spot the mistake when helping novices, without actually fixing >> it. >> >> (Of course requiring an "as" clause would solve all of this, but nobody >> wants that...) > > So you don't teach people to spell it as "except Exception". You teach > them to look up the actual exception type they want to catch. That > already happens, but I've seen quite a bit of code from my own > students where the shorthand of "except:" is so tempting that it gets > in. If that came straight back with SyntaxError, they'd be forced to > put _something_ in, and would be more likely to put in a useful > exception type. And you don't think they'd quickly learn that the way to keep being lazy is to write "except Exception: pass"? That seems overly optimistic to me, but then you're the one teaching classes, so I'll defer to you. (By the way, even "except TheRightException: pass" is going to be just as bad 90% of the time; they'll still be posting to StackOverflow asking "why my function does nothing and just returns?") >> Cons: >> * May make it harder to write Py2/Py3-compatible code. In Py2, >> "except:" will catch old-style-class exceptions. >> >> >> If the C API docs are right and exceptions must be derived from >> BaseException even in 2.6, they can't be old-style class instances in the >> first place. (And if anyone's trying to write 2.5/3.x code, they still have >> to deal with string exceptions, not to mention that they can't get the value >> of any exceptions they handle, so they've got much bigger problems.) > > Yeah, the wide-span compatibility issue is always going to be there. > But given that the docs aren't quite right here, this is an issue with > 2.7/3.x code. I agree that 2.5 can be pretty much ignored. 2.6, maybe not (a huge percentage of the libs on PyPI right now are 2.6+/3.3+), but it sounds like 2.6 and 2.7 are pretty much identical as far as exceptions anyway? >> Even if that's not right, what problems do old-style classes cause that >> new-style-but-not-BaseException-derived classes don't? > > I don't know. I've never consciously raised or caught an old-style > class. If I unwittingly did, there'd be no problem to any of my code > if it suddenly started inheriting from Exception or StandardError. I suspect the issue isn't so much consciously raising them, as, say, conditionally using different third-party libs if they exist (or even using user-written plugins), and one of them raises an old-style class instance, and your app used to catch it and now it doesn't, and you don't notice it until one of your users complains... > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Sat Apr 11 15:44:23 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Apr 2015 23:44:23 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> Message-ID: On Sat, Apr 11, 2015 at 11:16 PM, Andrew Barnert wrote: > That worries me a bit. If the docs say strings can be raised in 2.7, even if CPython doesn't allow it, I think you pretty much have to survey every implementation and make sure none of them allow it either before you can feel safe saying it's _just_ a doc problem... > >> 3) I have absolutely no idea about string exceptions - I do my best to >> avoid them. > > Well, yeah, but if they really do still exist in 2.7 or even 2.6, I think that would count as a strike against removing bare except. > I don't have a pile of other Python implementations handy, so I'll defer to someone else (Steven D'Aprano always seems to have a few dozen lying around, and might even have a handy script for running something on all of them and comparing the results!). >>>>> class OldException: pass >> ... >>>>> raise OldException >> Traceback (most recent call last): >> File "", line 1, in >> __main__.OldException: <__main__.OldException instance at 0x7f0e6c89e1b8> >> >> So that one probably needs a docs change too, although it's not as >> serious a problem. > > Maybe it means you can't raise them through the public C API, but you can still raise them through Python (or private C functions, presumably)? > That would be an *extremely* weird situation, and could probably be called a bug. But I don't write C extensions, so again, I'll defer to someone else on this point. > And you don't think they'd quickly learn that the way to keep being lazy is to write "except Exception: pass"? > > (By the way, even "except TheRightException: pass" is going to be just as bad 90% of the time; they'll still be posting to StackOverflow asking "why my function does nothing and just returns?") > It's always possible to write bad code. If the bad form is much shorter and more natural/discoverable than the good, then a lot more people will write bad code than if they're on par with each other. For instance, in Py2, it's very easy to write code that's Unicode-unaware simply by using the obvious string literal style everywhere; it runs just fine as long as everything's ASCII, and then it breaks on non-ASCII text. In Py3, you're forced up-front to differentiate between Unicode and byte strings, and the obvious string literal style gives you a Unicode string, so people take the obvious route and have proper Unicode handling. try: input = raw_input except NameError: pass name = input("Enter your name: ") print("REPORT FOR " + name.upper()) rosuav at sikorsky:~$ python2 ucase.py Enter your name: Beg?m G?nceler REPORT FOR BEG?M G?NCELER rosuav at sikorsky:~$ python3 ucase.py Enter your name: Beg?m G?nceler REPORT FOR BEG?M G?NCELER It's still possible for your code to be too naive, but you have a *much* better chance that simplistic code is Unicode-aware in Py3 than in Py2. Same with exception handling. Sure, you could still have "except ValueError: pass" and then wonder why your code isn't doing anything (or, worse, you create a completely different and hard-to-debug error elsewhere, because your function now returns None instead of something else - seen that happen a few times), but that's a lot less subtle than "why does my program go weird when I press Ctrl-C?" or "why does sys.exit(0) not work here?", which you'll get from a bare except. >>> Even if that's not right, what problems do old-style classes cause that >>> new-style-but-not-BaseException-derived classes don't? >> >> I don't know. I've never consciously raised or caught an old-style >> class. If I unwittingly did, there'd be no problem to any of my code >> if it suddenly started inheriting from Exception or StandardError. > > I suspect the issue isn't so much consciously raising them, as, say, conditionally using different third-party libs if they exist (or even using user-written plugins), and one of them raises an old-style class instance, and your app used to catch it and now it doesn't, and you don't notice it until one of your users complains... > That would be a nuisance, yes. But hopefully that can be deflected back onto the third-party lib/user-written plugin, basically by saying: What's new in MyLibrary v5.7.0: * Support for Python 3.6 * Old-style class exceptions are no longer supported. If your plugin defines custom exceptions, be sure that they derive from Exception or similar. * other changes, yada yada Every document I've ever seen has taken one of two attitudes toward old-style classes. Either they should be considered deprecated, and every class you define should explicitly subclass something; or they're a really cool micro-optimization with a few fancy flexibilities that new-style classes don't have, but which come up only in the most obscure cases. If a library has to say "old-style class exceptions can't be supported now due to Python 3.6 compatibility requirements", how many people would actually be up in arms? It's a VERY small removal of backward compat in favour of forward compat and cleaner code, just like abolishing string exceptions was. However, if it turns out there's some distinct advantage to having exceptions outside of the primary hierarchy, then I'll withdraw the suggestion, at least until post-2020 when Py2 support ends (and probably longer). ChrisA From steve at pearwood.info Sat Apr 11 16:59:57 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 12 Apr 2015 00:59:57 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> Message-ID: <20150411145956.GI5760@ando.pearwood.info> On Sat, Apr 11, 2015 at 10:39:04PM +1000, Chris Angelico wrote: > On Sat, Apr 11, 2015 at 10:25 PM, Andrew Barnert wrote: > > On Apr 11, 2015, at 04:29, Chris Angelico wrote: > > > > A bare "except:" clause catches any exception. As of Python 3.0, this > > can be spelled more explicitly as "except BaseException:", and AFAIK > > the only difference between the two is that someone might rebind > > BaseException. > > > > String exceptions were abolished during the 2.x line, without causing > > major upheaval. > > > > > > I couldn't remember when this happened, so I look at the docs... But 2.7 > > (https://docs.python.org/2.7/reference/executionmodel.html#exceptions) still > > say "Exceptions can also be identified by strings, in which case the except > > clause is selected by object identity." Do the docs need to be fixed? > > 1) Yes, sounds like it. > 2) By identity? Not by equality?? > 3) I have absolutely no idea about string exceptions - I do my best to > avoid them. Good plan :-) String exceptions were noisily deprecated in 2.5 and removed in 2.6, but they were obsolete for a long time before that. I don't think they were really common since the Python 1.5 days. And yes, they matched by identity! That was a common source of bugs back in 1.5 days. Python 2.7 does allow you to raise from arbitrary old-style classes: [steve at ando ~]$ python2.7 -c "class YeOldeClass: pass > raise YeOldeClass" Traceback (most recent call last): File "", line 2, in __main__.YeOldeClass: <__main__.YeOldeClass instance at 0xb7f2d1cc> but new style classes must be derived from BaseException in both 2.7 and 3.x, so people migrating from 2.x to 3.x will have to change their exceptions to inherit directly or indirectly from BaseException. This will not change that. * In 3.x only code, bare except is equivalent to `except BaseException`; * in 2.x only code, they are not the same; * but if you are migrating to 3.x, or have 2+3 hybrid code, you cannot use non-BaseException exceptions, which means that in hybrid code a bare except is *effectively* equivalent to `except BaseException`. So I don't think this proposed change would add any significant burden to migration. You still have to make sure all your exceptions are actual exceptions. I think that a bare except is an attractive nuisance. Nearly always it should be catching a specific exception, or at most Exception. Very rarely you want to catch BaseException. I think that Python would be better off without bare except clauses, having to explicitly write Exception or (more rarely) BaseException is a good thing. I can see four objections: (1) In the interactive interpreter, sometimes you don't care about being careful. Being able to just catch everything with a bare except is fine for interactive use. (I do so myself.) Response: I don't think much of this argument from laziness. If you're using the II a lot, you should have a startup file. Just define E = Exception in your startup file and then you can say "except E:" which is almost as convenient. Or use tab completion (enabled by default from 3.4 onwards). We shouldn't allow convenience in the interactive interpreter be an excuse for keeping a feature which gets misused as often as bare except does. (2) Should 2to3 automatically translate "except:" to "except Exception:" or "except BaseException:"? The second is the literal translation, but the first is what the programmer probably intended. Probably, but not always. Response: I think the 2to3 fixer will have to use BaseException, even though normally we recommend to catch Exception. If the fixer uses Exception, some code that was working will break (it now catches too little), but some code that was broken (it caught too much) will be less broken. If the fixer uses BaseException, working code will stay working, and broken code will stay broken. 2to3 isn't responsible for fixing your bugs, and it certainly shouldn't introduce bugs if it can avoid it. Hence, it must transform bare except to catching BaseException. (Maybe it can ask the programmer?) (3) Bare excepts have their uses and they aren't all bad. We should keep them. Response: Yeah, well, that's like just your opinion! *wink* I guess if you mostly work with good-quality mature code written by experienced Python developers who never use bare except clauses except when they really are needed, you may not think this is a problem to be solved. Bare excepts aren't *all* bad, and if they are abused occasionally, oh well, any feature can be abused. If, on the other hand, you mostly work with variable-quality code written by inexperienced, lazy, or just plain poor programmers who think that tossing a bare except around everything makes the code bug-free, then you might have a different opinion, and you might curse bare except as an attractive nuisance and want to see it die. (4) At this late stage in Python's development we shouldn't remove even a dubious feature. Every little incompatibility between 2.7 and 3.x just slows down migration. Response: I'm sympathetic to this argument, but I don't want to see "slows down migration" be used to block removal of mistakes. Even C removes features eventually (although on a much slower timescale than Python's relatively fast development cycle). But perhaps we don't have to remove them *right now*. We could deprecate them, and leave them deprecated indefinitely, for eventual removal in Python 5000 or something. > > Meanwhile, in the C API > > (https://docs.python.org/2.7/c-api/exceptions.html#string-exceptions), it > > says "Changed in version 2.6: All exceptions to be raised or caught must be > > derived from BaseException." I think that one is correct, but until I sit > > down at a bigger screen where I can browse the source or run an interpreter, > > I'm not sure. I think that means that the C API is now fixed so that all exceptions raised from builtins are now real exceptions, not random classes and certainly not strings. But it doesn't affect the Python API. > > Pros: > > * Remove the attractive nuisance of "hmm, my code's throwing an error > > that I don't recognize, I'll use try/except" and just catching > > everything. > > > > > > But if you just teach people "spell that as except BaseException:" (or, more > > likely, "... as except Exception:"), it's still the exact same attractive > > nuisance. And, because "except:" is almost always a red flag in novice code, > > but "except Exception:" is occasionally reasonable, you're making it a tiny > > bit harder to spot the mistake when helping novices, without actually fixing > > it. > > > > (Of course requiring an "as" clause would solve all of this, but nobody > > wants that...) > > So you don't teach people to spell it as "except Exception". You teach > them to look up the actual exception type they want to catch. That > already happens, but I've seen quite a bit of code from my own > students where the shorthand of "except:" is so tempting that it gets > in. If that came straight back with SyntaxError, they'd be forced to > put _something_ in, and would be more likely to put in a useful > exception type. That's the point. We cannot force people to catch the correct exception. We might not even know the correct exceptions to catch! But bare excepts are an attractive nuisance because (1) they're attractive because they're so easy (less typing is always good!) and (2) sometimes they're a bug magnet. We can't do anything about (2), but if people have to use an exception type, they may be less gung-ho about hiding bugs instead of fixing them ("except Exception is too much typing, I guess I'll just have to fix my code instead..." *wink*). -- Steve From steve at pearwood.info Sat Apr 11 17:15:08 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 12 Apr 2015 01:15:08 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> Message-ID: <20150411151507.GJ5760@ando.pearwood.info> On Sat, Apr 11, 2015 at 11:44:23PM +1000, Chris Angelico wrote: > Every document I've ever seen has taken one of two attitudes toward > old-style classes. Either they should be considered deprecated, and > every class you define should explicitly subclass something; or > they're a really cool micro-optimization with a few fancy > flexibilities that new-style classes don't have, but which come up > only in the most obscure cases. I wouldn't necessarily believe that classic classes are still faster than new-style classes. I think it was certainly true back in the 2.2 and 2.3 days, but probably not in 2.7. In any case, I'd like to see the benchmarks demonstrating a speed advantage to classic classes before believing that they are an optimization in practice. As far as I can see, the only things classic classes can do that new-style classes don't are: (1) Per-instance overriding of dunder methods on the instance. (I'm sure there's a Design Pattern name for this, but I can never remember it.) You can override regular methods for a particular instance by giving it an instance attribute that is a bound method: class Parrot: def speak(self): return "Want a cracker!" polly = Parrot() polly.speak = types.MethodType(lambda self: "Who's a pretty boy?", polly) but this doesn't work with dunders in new-style classes. In classic classes, it does. (2) Automatic delegation including dunder methods. Both of these may be important, but if you are migrating to 3.x you are going to lose them anyway, so you have to deal with it somehow. (Probably by crying a lot.) -- Steve From rosuav at gmail.com Sat Apr 11 17:21:03 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 12 Apr 2015 01:21:03 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: <20150411151507.GJ5760@ando.pearwood.info> References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> <20150411151507.GJ5760@ando.pearwood.info> Message-ID: On Sun, Apr 12, 2015 at 1:15 AM, Steven D'Aprano wrote: > On Sat, Apr 11, 2015 at 11:44:23PM +1000, Chris Angelico wrote: > >> Every document I've ever seen has taken one of two attitudes toward >> old-style classes. Either they should be considered deprecated, and >> every class you define should explicitly subclass something; or >> they're a really cool micro-optimization with a few fancy >> flexibilities that new-style classes don't have, but which come up >> only in the most obscure cases. > > I wouldn't necessarily believe that classic classes are still faster > than new-style classes. I think it was certainly true back in the 2.2 > and 2.3 days, but probably not in 2.7. In any case, I'd like to see the > benchmarks demonstrating a speed advantage to classic classes before > believing that they are an optimization in practice. Not sure if there's a speed advantage or not, but apparently there's a size advantage: http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see/ Though the difference doesn't apply to instances - a new-style class instance is marginally smaller than its compatriot - and the difference would be dwarfed by actual content anyway, in a typical program. Hence "micro-optimization". ChrisA From mal at egenix.com Sat Apr 11 17:29:43 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 11 Apr 2015 17:29:43 +0200 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: <20150411151507.GJ5760@ando.pearwood.info> References: <76937829-00CC-4C9C-88EF-F5322B5B13E3@yahoo.com> <20150411151507.GJ5760@ando.pearwood.info> Message-ID: <55293DE7.1050304@egenix.com> On 11.04.2015 17:15, Steven D'Aprano wrote: > On Sat, Apr 11, 2015 at 11:44:23PM +1000, Chris Angelico wrote: > >> Every document I've ever seen has taken one of two attitudes toward >> old-style classes. Either they should be considered deprecated, and >> every class you define should explicitly subclass something; or >> they're a really cool micro-optimization with a few fancy >> flexibilities that new-style classes don't have, but which come up >> only in the most obscure cases. > > I wouldn't necessarily believe that classic classes are still faster > than new-style classes. I think it was certainly true back in the 2.2 > and 2.3 days, but probably not in 2.7. In any case, I'd like to see the > benchmarks demonstrating a speed advantage to classic classes before > believing that they are an optimization in practice. See here (slide 39ff): http://www.egenix.com/library/presentations/PyCon-UK-2014-When-performance-matters/ https://github.com/egenix/when-performance-matters Instance attribute lookups are faster on old-style classes. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 11 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From travis.a.everett at gmail.com Sat Apr 11 17:40:04 2015 From: travis.a.everett at gmail.com (Travis Everett) Date: Sat, 11 Apr 2015 10:40:04 -0500 Subject: [Python-ideas] notify exception during raise (once magic properties are set) In-Reply-To: References: <21CDB9C8-55F9-47AF-A564-C2FA7584DC6D@yahoo.com> Message-ID: Thanks for the addendum--I suspected a mix of performance and the difficulty of handling potential failures at that point would be the ultimate answer to why the exception is already cut out of the loop there. Not that the dumber __raised__() version wouldn't have the same problem, but its behavior in case of failure could be straightforward. An idea I had last night led to some progress this morning. BaseException's with_traceback(tb) method already sets some thematic precedent for users being able to cause new exceptions while setting values within the raise process, (though unfortunately it looks like the traceback it sets gets overwritten by the new one set in C during the raise.) I realize that this is just a thematic precedent, since they'd be caused at very different places. For now I've replaced my previous workaround with a with_cause(c) method which is at least a step in the right direction for handling my specific case in a way that feels roughly congruent with existing idioms, makes sense in that context, and cleans up code using the exceptions; __cause__ still gets overwritten by the C code, but this doesn't cause trouble since we know what this object will be at raise time. This wouldn't suffice if I needed access to __traceback__ at this point. RE the question in your first response on patch benchmarking: I'm new to the list and have no previous contributions, but there's a first time for everything; if the information on doing so is readily available I should be fine, but otherwise I'd need some direction. On Fri, Apr 10, 2015 at 5:07 PM, Andrew Barnert wrote: > On Apr 10, 2015, at 14:39, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > > On Apr 10, 2015, at 12:54, Travis Everett > wrote: > > Hi all, > > I have a specialized use case for exceptions which need to analyze their > __cause__ property before they're fully initialized. The property isn't set > until after init, and (forgive my ignorance of Python's source!) appears to > be done at the C level in a way that it can't be observed with a property > setter or __setattr__. I currently create a method to do this post-init > work as a workaround, and push the burden of making sure it gets called on > the code using the exceptions--but this makes the exceptions idiomatic and > error-prone. > > A magic method called once all of the special exception properties are > already set would suffice. The exception could define something like > __raised__(self) to react to the new values of those properties. > > A more comprehensive approach might be a __raise__(self, cause, context, > traceback...) method which has full control over setting the properties, > but I assume there are good reasons why the usual channels (setters and > __setattr__) are cut out of the loop. > > > I'd assume the only reasons are to make the code a bit simpler and a bit > more efficient. > > > Actually, on second though, it's not _quite_ that simple. > > First, settings args, __traceback__ etc. are no longer hookable; it seems > like you'd want those to be as well, for consistency? So that's a few more > functions to change. > > Second, the API functions for setting cause and context (and the private > API function for automatically setting context) can't possibly fail > anywhere, so they have no return value. Changing them to go through SetAttr > means they now can (your __setattr__ could raise anything it wants--or even > creating the 1 object to store in __suppress_context__ could fail). Should > we just swallow and ignore any such exceptions when called from the C error > API? (C code that uses the exception type API or the object API will of > course continue to get exceptions, as will Python code, it's just the error > machinery that would ignore errors from munging exceptions to be raised.) > > Adding some new __raise__ or __raised__ special-purpose protocol doesn't > seem like it would be any simpler or more efficient than just changing the > existing APIs to access the attributes via SetAttr, which would give you > what you want without being a special case. > > Also notice that there's an open PEP ( > https://www.python.org/dev/peps/pep-0490/) to make the C API for raising > exceptions automatically set the context instead of forcing it to be done > manually (with a private API function), as a few builtin and stdlib > functions do. So it seems safer to change things in a way that would work > equally well with the current APIs, with that change, and with all > reasonable alternatives to that change--and again, using SetAttr seems like > the simplest way to make sure of that. > > But either way, it's a relatively local and simple change in > Objects/exceptions.c. > > I suspect the big thing you'd need to get this accepted is to show that > the performance cost is negligible. Do you know how to run appropriate > benchmarks and present the results if someone else writes the patch? > > I was encouraged to bring discussion here (to see if there's any > traction) after opening an enhancement request ( > http://bugs.python.org/issue23902). > > > > Cheers, > Travis > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Apr 11 21:18:25 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 11 Apr 2015 15:18:25 -0400 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: Message-ID: On 4/11/2015 7:29 AM, Chris Angelico wrote: > A bare "except:" clause catches any exception. As of Python 3.0, this > can be spelled more explicitly as "except BaseException:", and AFAIK > the only difference between the two is that someone might rebind > BaseException. > > String exceptions were abolished during the 2.x line, without causing > major upheaval. The deprecation and ultimate abolition of the bare > except syntax would have less consequence than that, as it would > merely require a mechanical transformation to "except BaseException:", > without needing any corresponding changes at raise site. > Thoughts? * There are about 20 bare excepts in idlelib and an issue (15313) to remove them by adding the missing class or tuple. I have looked at a few and found it is hard to determine what should be added. I may eventually just add Exception if I cannot be more specific (or general). * 'except:' is easily caught by a code checker, or reported by grep. This works both ways: don't need to change the language, but equally easy to find to change if the language is changed. * As Steven said, there could be a long deprecation period. -- Terry Jan Reedy From abarnert at yahoo.com Sun Apr 12 03:15:13 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 11 Apr 2015 18:15:13 -0700 Subject: [Python-ideas] notify exception during raise (once magic properties are set) In-Reply-To: References: <21CDB9C8-55F9-47AF-A564-C2FA7584DC6D@yahoo.com> Message-ID: <8607829D-6622-47E3-A09B-51A55F2CAC91@yahoo.com> > On Apr 11, 2015, at 08:40, Travis Everett wrote: > > Thanks for the addendum--I suspected a mix of performance and the difficulty of handling potential failures at that point would be the ultimate answer to why the exception is already cut out of the loop there. Not that the dumber __raised__() version wouldn't have the same problem, but its behavior in case of failure could be straightforward. Is it straightforward? When e.__raised__ raises, do you want to just swallow it and raise e anyway, or raise the exception raised by e.__raised__ with e as a context, or lose e and just raise the exception raised by e.__raised__? (Or you could just punt like C++ and say that if an exception type itself raises an exception in a place that's hard to deal with, the program terminates. :) Remember the possibility that what e.__raised__ raised may be a MemoryError, which always makes things extra fun. You have pretty much the same choices in setting the __cause__ or __context__; some of the answers may turn out to be harder to implement there than with __raised__, but instead of just assuming that means the ideal version of your proposal won't work, it's probably better to decide what behavior is actually right, and then see if it turns out to be impossible. So... Which one do you think is right in each case? > An idea I had last night led to some progress this morning. BaseException's with_traceback(tb) method already sets some thematic precedent for users being able to cause new exceptions while setting values within the raise process, (though unfortunately it looks like the traceback it sets gets overwritten by the new one set in C during the raise.) I realize that this is just a thematic precedent, since they'd be caused at very different places. > > For now I've replaced my previous workaround with a with_cause(c) method which is at least a step in the right direction for handling my specific case in a way that feels roughly congruent with existing idioms, makes sense in that context, and cleans up code using the exceptions; __cause__ still gets overwritten by the C code, but this doesn't cause trouble since we know what this object will be at raise time. This wouldn't suffice if I needed access to __traceback__ at this point. > > RE the question in your first response on patch benchmarking: I'm new to the list and have no previous contributions, but there's a first time for everything; if the information on doing so is readily available I should be fine, but otherwise I'd need some direction. > >>> On Fri, Apr 10, 2015 at 5:07 PM, Andrew Barnert wrote: >>>> On Apr 10, 2015, at 14:39, Andrew Barnert wrote: >>>> >>>> On Apr 10, 2015, at 12:54, Travis Everett wrote: >>>> >>>> Hi all, >>>> >>>> I have a specialized use case for exceptions which need to analyze their __cause__ property before they're fully initialized. The property isn't set until after init, and (forgive my ignorance of Python's source!) appears to be done at the C level in a way that it can't be observed with a property setter or __setattr__. I currently create a method to do this post-init work as a workaround, and push the burden of making sure it gets called on the code using the exceptions--but this makes the exceptions idiomatic and error-prone. >>>> >>>> A magic method called once all of the special exception properties are already set would suffice. The exception could define something like __raised__(self) to react to the new values of those properties. >>>> >>>> A more comprehensive approach might be a __raise__(self, cause, context, traceback...) method which has full control over setting the properties, but I assume there are good reasons why the usual channels (setters and __setattr__) are cut out of the loop. >>> >>> I'd assume the only reasons are to make the code a bit simpler and a bit more efficient. >> >> Actually, on second though, it's not _quite_ that simple. >> >> First, settings args, __traceback__ etc. are no longer hookable; it seems like you'd want those to be as well, for consistency? So that's a few more functions to change. >> >> Second, the API functions for setting cause and context (and the private API function for automatically setting context) can't possibly fail anywhere, so they have no return value. Changing them to go through SetAttr means they now can (your __setattr__ could raise anything it wants--or even creating the 1 object to store in __suppress_context__ could fail). Should we just swallow and ignore any such exceptions when called from the C error API? (C code that uses the exception type API or the object API will of course continue to get exceptions, as will Python code, it's just the error machinery that would ignore errors from munging exceptions to be raised.) >> >>> Adding some new __raise__ or __raised__ special-purpose protocol doesn't seem like it would be any simpler or more efficient than just changing the existing APIs to access the attributes via SetAttr, which would give you what you want without being a special case. >>> >>> Also notice that there's an open PEP (https://www.python.org/dev/peps/pep-0490/) to make the C API for raising exceptions automatically set the context instead of forcing it to be done manually (with a private API function), as a few builtin and stdlib functions do. So it seems safer to change things in a way that would work equally well with the current APIs, with that change, and with all reasonable alternatives to that change--and again, using SetAttr seems like the simplest way to make sure of that. >>> >>> But either way, it's a relatively local and simple change in Objects/exceptions.c. >>> >>> I suspect the big thing you'd need to get this accepted is to show that the performance cost is negligible. Do you know how to run appropriate benchmarks and present the results if someone else writes the patch? >>> >>>> I was encouraged to bring discussion here (to see if there's any traction) after opening an enhancement request (http://bugs.python.org/issue23902). >>> >>> >>>> Cheers, >>>> Travis >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From waultah at gmail.com Sun Apr 12 09:57:25 2015 From: waultah at gmail.com (Riley Banks) Date: Sun, 12 Apr 2015 10:57:25 +0300 Subject: [Python-ideas] Define __getitem__ for PurePath Message-ID: Greetings. I think it'd be nice to have the ability to subscript/slice instances of pathlib.PurePath and its subclasses e.g. path = Path() assert path[n:m:k] == type(path)(*path.parts[n:m:k]) Maybe it's worth adding some check with custom exception thrown in case of invalid index/slice. What do you think? I'll open new issue and submit a patch if you approve of this... -------------- next part -------------- An HTML attachment was scrubbed... URL: From travis.a.everett at gmail.com Sun Apr 12 18:39:30 2015 From: travis.a.everett at gmail.com (Travis Everett) Date: Sun, 12 Apr 2015 11:39:30 -0500 Subject: [Python-ideas] notify exception during raise (once magic properties are set) In-Reply-To: <8607829D-6622-47E3-A09B-51A55F2CAC91@yahoo.com> References: <21CDB9C8-55F9-47AF-A564-C2FA7584DC6D@yahoo.com> <8607829D-6622-47E3-A09B-51A55F2CAC91@yahoo.com> Message-ID: There are certainly still questions about appropriate behavior. I think the least-surprising result of a failure in e.__raised__ is to raise with context, but I realize I'm naive about the implementation-level implications of doing so. I thought twice about using that word, but by calling __raised__ straightforward, I meant that a failure in the __raise__ version creates more ambiguity in regard to what those properties will contain. Does it keep only the properties successfully set before failure and attempt no recovery? Does it try to make lemonade by keeping the successful properties, but set default values for the others? Does it overwrite all values with defaults as if the magic method wasn't defined? I'd say the first is least surprising because it lacks magic, but it means a failure can break the cause/context chain; these all seem quite surprising in their own ways. On Sat, Apr 11, 2015 at 8:15 PM, Andrew Barnert wrote: > On Apr 11, 2015, at 08:40, Travis Everett > wrote: > > Thanks for the addendum--I suspected a mix of performance and the > difficulty of handling potential failures at that point would be the > ultimate answer to why the exception is already cut out of the loop there. > Not that the dumber __raised__() version wouldn't have the same problem, > but its behavior in case of failure could be straightforward. > > > Is it straightforward? When e.__raised__ raises, do you want to just > swallow it and raise e anyway, or raise the exception raised by > e.__raised__ with e as a context, or lose e and just raise the exception > raised by e.__raised__? (Or you could just punt like C++ and say that if an > exception type itself raises an exception in a place that's hard to deal > with, the program terminates. :) Remember the possibility that what > e.__raised__ raised may be a MemoryError, which always makes things extra > fun. > > You have pretty much the same choices in setting the __cause__ or > __context__; some of the answers may turn out to be harder to implement > there than with __raised__, but instead of just assuming that means the > ideal version of your proposal won't work, it's probably better to decide > what behavior is actually right, and then see if it turns out to be > impossible. > > So... Which one do you think is right in each case? > > An idea I had last night led to some progress this morning. > BaseException's with_traceback(tb) method already sets some thematic > precedent for users being able to cause new exceptions while setting values > within the raise process, (though unfortunately it looks like the traceback > it sets gets overwritten by the new one set in C during the raise.) I > realize that this is just a thematic precedent, since they'd be caused at > very different places. > > For now I've replaced my previous workaround with a with_cause(c) method > which is at least a step in the right direction for handling my specific > case in a way that feels roughly congruent with existing idioms, makes > sense in that context, and cleans up code using the exceptions; __cause__ > still gets overwritten by the C code, but this doesn't cause trouble since > we know what this object will be at raise time. This wouldn't suffice if I > needed access to __traceback__ at this point. > > RE the question in your first response on patch benchmarking: I'm new to > the list and have no previous contributions, but there's a first time for > everything; if the information on doing so is readily available I should be > fine, but otherwise I'd need some direction. > > On Fri, Apr 10, 2015 at 5:07 PM, Andrew Barnert > wrote: > >> On Apr 10, 2015, at 14:39, Andrew Barnert < >> abarnert at yahoo.com.dmarc.invalid> wrote: >> >> On Apr 10, 2015, at 12:54, Travis Everett >> wrote: >> >> Hi all, >> >> I have a specialized use case for exceptions which need to analyze their >> __cause__ property before they're fully initialized. The property isn't set >> until after init, and (forgive my ignorance of Python's source!) appears to >> be done at the C level in a way that it can't be observed with a property >> setter or __setattr__. I currently create a method to do this post-init >> work as a workaround, and push the burden of making sure it gets called on >> the code using the exceptions--but this makes the exceptions idiomatic and >> error-prone. >> >> A magic method called once all of the special exception properties are >> already set would suffice. The exception could define something like >> __raised__(self) to react to the new values of those properties. >> >> A more comprehensive approach might be a __raise__(self, cause, context, >> traceback...) method which has full control over setting the properties, >> but I assume there are good reasons why the usual channels (setters and >> __setattr__) are cut out of the loop. >> >> >> I'd assume the only reasons are to make the code a bit simpler and a bit >> more efficient. >> >> >> Actually, on second though, it's not _quite_ that simple. >> >> First, settings args, __traceback__ etc. are no longer hookable; it seems >> like you'd want those to be as well, for consistency? So that's a few more >> functions to change. >> >> Second, the API functions for setting cause and context (and the private >> API function for automatically setting context) can't possibly fail >> anywhere, so they have no return value. Changing them to go through SetAttr >> means they now can (your __setattr__ could raise anything it wants--or even >> creating the 1 object to store in __suppress_context__ could fail). Should >> we just swallow and ignore any such exceptions when called from the C error >> API? (C code that uses the exception type API or the object API will of >> course continue to get exceptions, as will Python code, it's just the error >> machinery that would ignore errors from munging exceptions to be raised.) >> >> Adding some new __raise__ or __raised__ special-purpose protocol doesn't >> seem like it would be any simpler or more efficient than just changing the >> existing APIs to access the attributes via SetAttr, which would give you >> what you want without being a special case. >> >> Also notice that there's an open PEP ( >> https://www.python.org/dev/peps/pep-0490/) to make the C API for raising >> exceptions automatically set the context instead of forcing it to be done >> manually (with a private API function), as a few builtin and stdlib >> functions do. So it seems safer to change things in a way that would work >> equally well with the current APIs, with that change, and with all >> reasonable alternatives to that change--and again, using SetAttr seems like >> the simplest way to make sure of that. >> >> But either way, it's a relatively local and simple change in >> Objects/exceptions.c. >> >> I suspect the big thing you'd need to get this accepted is to show that >> the performance cost is negligible. Do you know how to run appropriate >> benchmarks and present the results if someone else writes the patch? >> >> I was encouraged to bring discussion here (to see if there's any >> traction) after opening an enhancement request ( >> http://bugs.python.org/issue23902). >> >> >> >> Cheers, >> Travis >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Mon Apr 13 00:39:45 2015 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 13 Apr 2015 08:39:45 +1000 Subject: [Python-ideas] Proposal: Abolition of bare except clauses In-Reply-To: References: Message-ID: <20150412223945.GA71875@cskk.homeip.net> For the record, as a Python 2/3 user of the occasional bare "except:" I was against this over on python-list, but Chris and Steven have convinced me otherwise. I'm now +0.5 and will probably be +1 in time. I was using a few bare excepts in boundary code that needed to plough on in the face of failures of a callable, but have been convinced that I would be better off with "except Exception:" most of the time, and I'm going to put some work into stripping out my bare "except:"s and mostly catching Exception and possibly reporting BaseException (and reraising) as a debugging/insight aid. Cheers, Cameron Simpson Ho, HaHa, Dodge, Parry, Spin, HA! THRUST! - Daffy Duck From rustompmody at gmail.com Mon Apr 13 07:21:05 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 13 Apr 2015 10:51:05 +0530 Subject: [Python-ideas] doc anomalies Message-ID: Hi all As a teacher who often ends up teaching python, sometimes I/my students encounter little anomalies [Bugs?? I hesitate to use the word given how often Ive seen a noob exclaim: "Compiler has a BUG" without knowing abc of the syntax ] So just a question (or 2): 1. In python3 please check help() followed by PACKAGES and DYNAMICFEATURES Do those help texts look helpful/meaningful? 2. Is this the right list for such questions? Rusi -------------- next part -------------- An HTML attachment was scrubbed... URL: From taleinat at gmail.com Mon Apr 13 11:25:39 2015 From: taleinat at gmail.com (Tal Einat) Date: Mon, 13 Apr 2015 12:25:39 +0300 Subject: [Python-ideas] doc anomalies In-Reply-To: References: Message-ID: Hi Rustom, On Mon, Apr 13, 2015 at 8:21 AM, Rustom Mody wrote: > Hi all > > As a teacher who often ends up teaching python, sometimes I/my students > encounter little anomalies > > [Bugs?? I hesitate to use the word given how often Ive seen a noob exclaim: > "Compiler has a BUG" without knowing abc of the syntax > ] > > So just a question (or 2): > 1. > In python3 please check > help() > followed by > PACKAGES > and > DYNAMICFEATURES > > Do those help texts look helpful/meaningful? Please specify the precise version of Python you are referencing (e.g. 3.4.2). I would be helpful if you didn't just throw out these questions, but also suggested improvements. > 2. > Is this the right list for such questions? This list is for ideas for the improvement of Python. Discussing significant changes to documentation and help resources can be relevant here, depending on their nature and breadth. - Tal From rustompmody at gmail.com Mon Apr 13 11:42:59 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 13 Apr 2015 15:12:59 +0530 Subject: [Python-ideas] doc anomalies In-Reply-To: References: Message-ID: On Mon, Apr 13, 2015 at 2:55 PM, Tal Einat wrote: > > Hi Rustom, > > On Mon, Apr 13, 2015 at 8:21 AM, Rustom Mody wrote: > > Hi all > > > > As a teacher who often ends up teaching python, sometimes I/my students > > encounter little anomalies > > > > [Bugs?? I hesitate to use the word given how often Ive seen a noob exclaim: > > "Compiler has a BUG" without knowing abc of the syntax > > ] > > > > So just a question (or 2): > > 1. > > In python3 please check > > help() > > followed by > > PACKAGES > > and > > DYNAMICFEATURES > > > > Do those help texts look helpful/meaningful? > > Please specify the precise version of Python you are referencing (e.g. 3.4.2). > Yeah 3.4.2 2.7.8 seems much the same > > I would be helpful if you didn't just throw out these questions, but > also suggested improvements. > As for DYNAMICFEATURES the text makes no sense (to me)... so dont know what to suggest As for PACKAGES I guess one could come up with a much cut-down and pointed version... if that's the intent > > 2. > > Is this the right list for such questions? > > This list is for ideas for the improvement of Python. Discussing > significant changes to documentation and help resources can be > relevant here, depending on their nature and breadth. Well in fact a bunch of my students and myself will be working on various sides of python internals. Dunno how much will finally be suitable for actual patches and how much just mucking around :-) So was wondering about the best sources of help/guidance with python internals From taleinat at gmail.com Mon Apr 13 13:19:19 2015 From: taleinat at gmail.com (Tal Einat) Date: Mon, 13 Apr 2015 14:19:19 +0300 Subject: [Python-ideas] doc anomalies In-Reply-To: References: Message-ID: On Mon, Apr 13, 2015 at 12:42 PM, Rustom Mody wrote: > On Mon, Apr 13, 2015 at 2:55 PM, Tal Einat wrote: >> >> I would be helpful if you didn't just throw out these questions, but >> also suggested improvements. >> > > As for DYNAMICFEATURES the text makes no sense (to me)... so dont know > what to suggest You could either suggest the complete removal of this section, or an improved version of it. > As for PACKAGES I guess one could come up with a much cut-down and > pointed version... if that's the intent Well, that depends on what the goal of that section is. From a quick read, it is a technical description of the "import" keyword and how the import mechanism works (it and IMPORTING are actually the same topic). I too find that somewhat surprising, since that's not the information I would be looking for when trying to read about "packages". Rather, I would expect an explanation of what packages are, and then perhaps how to create them and how they work. It seems to me that there is room for improvement here. >> > 2. >> > Is this the right list for such questions? >> >> This list is for ideas for the improvement of Python. Discussing >> significant changes to documentation and help resources can be >> relevant here, depending on their nature and breadth. > > Well in fact a bunch of my students and myself will be working on > various sides of python internals. > Dunno how much will finally be suitable for actual patches and how > much just mucking around :-) > So was wondering about the best sources of help/guidance with python internals Having students poke around the internals sounds great! python-list is always a good place to start, and definitely the right place for inquiries when you're not sure what the proper place is. In this case, I think you should write the core-mentorship list (see pythonmentors.com for details). It's been quite active lately and you are likely to receive helpful information, guidance and support for your initiative. - Tal From steve at pearwood.info Mon Apr 13 15:21:25 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 13 Apr 2015 23:21:25 +1000 Subject: [Python-ideas] doc anomalies In-Reply-To: References: Message-ID: <20150413132125.GB5663@ando.pearwood.info> On Mon, Apr 13, 2015 at 10:51:05AM +0530, Rustom Mody wrote: > In python3 please check > help() > followed by > PACKAGES > and > DYNAMICFEATURES > > Do those help texts look helpful/meaningful? PACKAGES certainly looks helpful and meaningful, if it were reached via IMPORTING instead :-) If I wanted to know about packages, and got five pages of text about importing, I would be rather confused and annoyed. I would like to see PACKAGES lead to a short description of packages and their structure, the special role of __init__.py and __main__.py, and, hmmm, I think they are called "namespace packages"? They're a fairly new feature, and I've never used them. It may also include "See also IMPORT." DYNAMICFEATURES is also useful, though less useful than it could be due to excessive jargon that isn't defined. What's a free variable? Can we see examples of what dynamic features can and cannot be used? I actually expected to see a lot more dynamic features discussed: - getattr, setattr, delattr - exec - eval - compile - introspection via the inspect module, dir and vars - Python equivalents to Java-like "reflection" and probably more, but that may just mean my idea of dynamic features are different to the author of that help text. > Is this the right list for such questions? Personally, I would have started by asking on the python-list list (comp.lang.python) and then gone straight to raising an issue on the bugtracker, if there was an issue to raise. But since we're here, a few comments come to mind for discussion... (1) Help topics seem to be case sensitive. That's not very helpful: help> packages no Python documentation found for 'packages' help> PACKAGES [masses of text] (2) Hitting the enter key with nothing on the help? prompt exits help. That's downright ugly. (3) Does help() have an "apropos" or "search" feature, for searching the bodies of help pages for keywords? -- Steve From rustompmody at gmail.com Mon Apr 13 16:17:09 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Mon, 13 Apr 2015 19:47:09 +0530 Subject: [Python-ideas] doc anomalies In-Reply-To: <20150413132125.GB5663@ando.pearwood.info> References: <20150413132125.GB5663@ando.pearwood.info> Message-ID: On Mon, Apr 13, 2015 at 6:51 PM, Steven D'Aprano wrote: > On Mon, Apr 13, 2015 at 10:51:05AM +0530, Rustom Mody wrote: > >> In python3 please check >> help() >> followed by >> PACKAGES >> and >> DYNAMICFEATURES >> >> Do those help texts look helpful/meaningful? > > PACKAGES certainly looks helpful and meaningful, if it were reached via > IMPORTING instead :-) > > If I wanted to know about packages, and got five pages of text about > importing, I would be rather confused and annoyed. > > I would like to see PACKAGES lead to a short description of packages and > their structure, the special role of __init__.py and __main__.py, and, > hmmm, I think they are called "namespace packages"? They're a fairly new > feature, and I've never used them. It may also include "See also > IMPORT." > > DYNAMICFEATURES is also useful, though less useful than it could be due > to excessive jargon that isn't defined. What's a free variable? Can we > see examples of what dynamic features can and cannot be used? > > I actually expected to see a lot more dynamic features discussed: > > - getattr, setattr, delattr > - exec > - eval > - compile > - introspection via the inspect module, dir and vars > - Python equivalents to Java-like "reflection" > > and probably more, but that may just mean my idea of dynamic features > are different to the author of that help text. > > >> Is this the right list for such questions? > But since we're here, a few comments come to mind for discussion... > > (1) Help topics seem to be case sensitive. That's not very helpful: > > help> packages > no Python documentation found for 'packages' > > help> PACKAGES > [masses of text] > > (2) Hitting the enter key with nothing on the help? prompt exits help. > That's downright ugly. > > (3) Does help() have an "apropos" or "search" feature, for searching the > bodies of help pages for keywords? Thanks Steven -- those are helpful points. > > Personally, I would have started by asking on the python-list list > (comp.lang.python) and then gone straight to raising an issue on the > bugtracker, if there was an issue to raise. > As I said, earlier, it looks as though a few students will be working with me on python-internals related projects. I was looking at some trivial bugs to fix to get my (our) toes wet. [One thing one learns as a wizened old programmer is that when we write a loop we try it on zero and one iteration before trying it on 1 million] Tal has suggested that I join the mentor list which I have done. From sffjunkie at gmail.com Tue Apr 14 14:14:55 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Tue, 14 Apr 2015 13:14:55 +0100 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version Message-ID: Call me a fool, but in the vein of the finest of half baked ideas, I present to you an idea that hasn't even decided on the shape of the tin yet. Using a pseudo random selection of ideas set out in various forums * python-dev * distutils-sig * python-ideas * import-sig I present to you, in the linked PDF, a possible Python future. Python 3.9.9 - The 'I Have A Dream' Version http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.pdf http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.xmind my ongoing thought experiment on a Python future I would like to see. Because, is it not written If you don't know where you're going, how will you know when you get there. Whether an infinite number of monkeys could shape this into reality may never be put to the test, but it's a start. Or it may not be. Or it may be a start and the end. The future's as tricksy as hobbitses sometimes. Highlights include * Python The - Executable - Language - Library - Tools - Ecosystem * Removing the notion of 'virtual' environments. Everything's just an environment. * Splitting the standard library in twain. stdlib = 'baselib' + 'extlib'. - baselib = Enough to start up a Python environment / install pip. - extlib = The rest of the current stdlib. Changes more frequently than the baselib. * A new tool called pippin to manage the extlib * Formalising what a package index is * Addressing a Python project's lifecycle. * Configuration handling * Removing the use of 'lib' suffixes from almost all packages. * Claiming any name in PyPi starting with 'py' And more. I have aimed for * Separation of concerns * Consistency in naming of tool commands and options. Your thoughts are welcome. Next steps are * Add all relevant PEPs - both proposed and accepted. * Add all modules to either baselib, extlib or deletelib * Firm up stdlib / environment / config file locations (system / user, Windows / Linux / OSX) * Create outline of pip / twine code to see what can be put into external packages for re-use / removal from pip. * Create a filesystem structure for projects and configuration files / cookiecutter templates to achieve this. * Enhance definition of tools and propose code structures. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Apr 14 15:01:05 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 14 Apr 2015 23:01:05 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: Message-ID: <20150414130104.GF5663@ando.pearwood.info> On Tue, Apr 14, 2015 at 01:14:55PM +0100, Simon Kennedy wrote: > Call me a fool, but in the vein of the finest of half baked ideas, I > present to you an idea that hasn't even decided on the shape of the tin yet. Well, you've certainly been meticulous :-) [...] > * Removing the notion of 'virtual' environments. Everything's just an > environment. What's the difference? > * Splitting the standard library in twain. stdlib = 'baselib' + 'extlib'. > - baselib = Enough to start up a Python environment / install pip. > - extlib = The rest of the current stdlib. Changes more frequently than > the baselib. Hmmm. That would have consequences. Are you okay with abandoning large parts of the corporate, government and education sectors? The thing is, one of the reasons that Python is so successful is that it is *batteries included*. That's been a selling point for 20 years. This means that using only the standard library, you can be productive and get real work done. This is important for people using a standard operating environment, or in restricted networks with no access to the internet, or in any scenario where people have to get approval before they can use a particular software package. Not everyone is able to just download arbitrary packages from the Internet and use them on the spur of the moment. With Python, they only need to get *one* package approved: "Can I download Python and use it?" It might take six weeks to get approval, but once you do, you have not just the core language, but a good 150+ solid, useful, powerful libraries (and a few not so good ones). With your Python 3.9.9, they get the core language, and a mechanism for downloading from PyPI, which they are not allowed to do without approval, on pain of dismissal. Or not able to do, because the corporate network blocks it. And don't even think about sneaking it in via USB key from home... So each and every package that needs to be downloaded from PyPI could take six weeks to get approval before they can run "pip install os". The thing is, the broader Python ecosystem is not just hackers on Linux with control over their own machines. It includes people in schools using locked-down SOEs, corporate users with restricted network access or paranoid legal/IT departments, people in developing nations with unreliable internet access, and more. > * A new tool called pippin to manage the extlib Does this actually exist, or have you picked it because the name is cool? (It is a cool name.) (There is already a pippin on PyPI and github. I don't know if they are the same project. The one of github claims to be experimental. Remember that most experiments fail.) > * Formalising what a package index is What do you mean? > * Addressing a Python project's lifecycle. Surely that will be different for every project? > * Configuration handling What about it? > * Removing the use of 'lib' suffixes from almost all packages. Why? > * Claiming any name in PyPi starting with 'py' That's going to be *really unpopular*. What is the purpose behind that? -- Steve From rosuav at gmail.com Tue Apr 14 15:44:05 2015 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 14 Apr 2015 23:44:05 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 10:14 PM, Simon Kennedy wrote: > * Splitting the standard library in twain. stdlib = 'baselib' + 'extlib'. > - baselib = Enough to start up a Python environment / install pip. > - extlib = The rest of the current stdlib. Changes more frequently than > the baselib. This could work, and answer Steven's objections, if what you have is three tiers: the two you describe, plus leaving PyPI as it already is. All you have to do is formalize a process for updating the extlib, such as: * Point releases of Python may make backward-compatible changes to the extlib (eg expand PEP 434 to cover more packages); or * extlib packages are duplicated onto PyPI, and can thus be updated/backported; or * extlib packages are distributed via a dedicated update mechanism, and get updated as an entire unit or something of the sort. Basically, the extlib and the baselib together comprise the "Python Standard Library", and you can depend on having all of its modules on all installations (subject to platform concerns of course). > * A new tool called pippin to manage the extlib Which might be the third option listed above. > * Formalising what a package index is Not sure what this would mean. Are you talking about turning PyPI into just one of many package indexes, the way deb and rpm repositories are distributed? > * Addressing a Python project's lifecycle. How? > * Configuration handling How? > * Removing the use of 'lib' suffixes from almost all packages. Breaks backward compatibility for minimal benefit. Why do it? > * Claiming any name in PyPi starting with 'py' I suppose this is part of putting the extlib there? IMO that shouldn't be necessary. Conflicts are won by the first package to get published. There's going to be STRONG pushback against backward-incompatible changes. Py2->Py3 is still transitioning, and we don't want to be eternally managing multiple codebases or warped compatibility code. Imagine how it'd look: try: from url.parse import unquote # Python 4 except ImportError: try: from urllib.parse import unquote # Python 3 except ImportError: from urllib import unquote # Python 2 You could bury it behind an "import twentyfour", but it's still a mess. And this is assuming there's identical functionality in the modules - otherwise you need even more shims. I would recommend focusing on incremental changes. If you want to see the stdlib divided such that more packages can be upgraded more frequently, that might be doable without breaking everything. Renaming "virtual environment" to simply "environment" and treating the base installation as "just another environment" ought to be doable too (though you'd have to do up a detailed proposal to be sure). The renamings probably just aren't worth doing, but maybe you could convince people to add alternate names and deprecate the old ones. There's no particular reason for it all to be packaged together into one monster proposal :) ChrisA From abarnert at yahoo.com Tue Apr 14 15:51:34 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 14 Apr 2015 06:51:34 -0700 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: Message-ID: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> On Apr 14, 2015, at 05:14, Simon Kennedy wrote: > > Call me a fool, but in the vein of the finest of half baked ideas, I present to you an idea that hasn't even decided on the shape of the tin yet. > > Using a pseudo random selection of ideas set out in various forums > > * python-dev > * distutils-sig > * python-ideas > * import-sig > > I present to you, in the linked PDF, a possible Python future. > > Python 3.9.9 - The 'I Have A Dream' Version > > http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.pdf > http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.xmind Most of this document seems so ambiguous that it's hard to know how to comment on it. For example, you've got a section called "Types" (which I picked arbitrarily because my PDF reader decided to start with the top right corner of the giant page 1), which lists the types as "int, bytes, string, float, list, tuple, named tuple, dict, ordered dict, set, frozenset, enum, class, module, generator, iterator, iterable, context manager, function". What does that mean? A good number of these are builtin types in 3.5; the most obvious implication is that the rest of them should become builtin types in 3.9.9. But what would it mean for namedtuple to be a type instead of a type factory? How can "class" be a type? If iterable becomes a concrete type, does that mean classes that implement the __iter__ method are no longer iterables, just "iterable-like types" (and likewise for context manager)? What would be the benefit of renaming "str" to "string"? Does the fact that you've left out bool and NoneType and method mean those will no longer be builtin types in 3.9.9? Some of your type names have spaces--major syntax change to the language, or typo? Or, if this doesn't mean any such thing, and is just an attempt to categorize the kinds of stuff that "type-related stuff" will have to deal with in 3.9.9, how is it any different from 3.5, which already has the same kinds stuff? Picking another part of the page at random, you see to have a stuff that goes into detail about specific paths for Windows vs. FHS/XDG. Does this mean you want to drop Mac support, remove the ability to configure and install into /opt/local or ~, etc.? If not, does it mean anything at all? Right above that, how do you expect to automatically work with every configuration file format, from .ini and .rc files to the Windows registry (but not Mac defaults...)? Does that mean you're going to provide some lowest-common-denominator API (a flat list of C-identifier-style string keys mapping to printable ASCII string values with no newlines seems to be about as flexible as you can get)? If not, what else could it mean to support every format? > > my ongoing thought experiment on a Python future I would like to see. Because, is it not written > > If you don't know where you're going, how will you know when you get there. > > Whether an infinite number of monkeys could shape this into reality may never be put to the test, but it's a start. > Or it may not be. Or it may be a start and the end. The future's as tricksy as hobbitses sometimes. > > Highlights include > > * Python The > - Executable > - Language > - Library > - Tools > - Ecosystem > > * Removing the notion of 'virtual' environments. Everything's just an environment. > > * Splitting the standard library in twain. stdlib = 'baselib' + 'extlib'. > - baselib = Enough to start up a Python environment / install pip. > - extlib = The rest of the current stdlib. Changes more frequently than the baselib. > > * A new tool called pippin to manage the extlib > > * Formalising what a package index is > > * Addressing a Python project's lifecycle. > > * Configuration handling > > * Removing the use of 'lib' suffixes from almost all packages. > > * Claiming any name in PyPi starting with 'py' > > And more. > > I have aimed for > > * Separation of concerns > * Consistency in naming of tool commands and options. > > Your thoughts are welcome. > > Next steps are > > * Add all relevant PEPs - both proposed and accepted. > * Add all modules to either baselib, extlib or deletelib > * Firm up stdlib / environment / config file locations (system / user, Windows / Linux / OSX) > * Create outline of pip / twine code to see what can be put into external packages for re-use / removal from pip. > * Create a filesystem structure for projects and configuration files / cookiecutter templates to achieve this. > * Enhance definition of tools and propose code structures. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sffjunkie at gmail.com Tue Apr 14 16:58:43 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Tue, 14 Apr 2015 07:58:43 -0700 (PDT) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <20150414130104.GF5663@ando.pearwood.info> References: <20150414130104.GF5663@ando.pearwood.info> Message-ID: On Tuesday, 14 April 2015 14:01:30 UTC+1, Steven D'Aprano wrote: > > > Well, you've certainly been meticulous :-) > > [...] > > * Removing the notion of 'virtual' environments. Everything's just an > > environment. > > What's the difference? > Not a lot ;-) Merely a reduction in concepts by 1. The question I asked myself when I thought about it is. Why 'virtual'? What's the conceptual difference between a standard Python install and a virtualenv (now that no-site-packages is the default.)? How does it help a Python user to distinguish between the two? Why do we have py.exe which finds a Python install but not a virtualenv? So why not merge the Python install and the virtual environment into a single concept - Python Environments Then we could have two tools 1. 'py' which finds and starts environments. (plus py could automatically find scripts in the bin / scripts / tools directories without having to add them to the path) 2. 'pyenv' which manages environments (create, list, show info, activate etc.) > > > * Splitting the standard library in twain. stdlib = 'baselib' + > 'extlib'. > > - baselib = Enough to start up a Python environment / install pip. > > - extlib = The rest of the current stdlib. Changes more frequently > than > > the baselib. > > Hmmm. That would have consequences. Are you okay with abandoning large > parts of the corporate, government and education sectors? > > The thing is, one of the reasons that Python is so successful is that it > is *batteries included*. That's been a selling point for 20 years. This > means that using only the standard library, you can be productive and > get real work done. This is important for people using a standard > operating environment, or in restricted networks with no access to the > internet, or in any scenario where people have to get approval before > they can use a particular software package. Not everyone is able to just > download arbitrary packages from the Internet and use them on the spur > of the moment. > > With Python, they only need to get *one* package approved: "Can I > download Python and use it?" It might take six weeks to get approval, > but once you do, you have not just the core language, but a good 150+ > solid, useful, powerful libraries (and a few not so good ones). > > With your Python 3.9.9, they get the core language, and a mechanism for > downloading from PyPI, which they are not allowed to do without > approval, on pain of dismissal. Or not able to do, because the corporate > network blocks it. And don't even think about sneaking it in via USB key > from home... So each and every package that needs to be downloaded from > PyPI could take six weeks to get approval before they can run "pip > install os". > > The thing is, the broader Python ecosystem is not just hackers on Linux > with control over their own machines. It includes people in schools > using locked-down SOEs, corporate users with restricted network access > or paranoid legal/IT departments, people in developing nations with > unreliable internet access, and more. > > The idea of the baselib + extlib is an attempt to address the desire to update parts of the stdlib faster than every Python release. You would still get all the batteries included. When installing a new version of Python the extlib would be installed along with the baselib just not along side the baselib i.e. it would be in a separate on disk structure. A new version of Python would be * python.exe + baselib * extlib on disk structure somewhere else The pippin tool which manages the extlib can pull in updates to the on disk repository in a batch as and when the maintainer wants. and the environment's maintainer can update the environment with the version they want. Installation from the extlib disk structure to the current environment is 'just a copy' operation. A difference then between a package on PyPi and in the extlib is only that the extlib is 'blessed' by the same people who have control of the Python releases. > > > * A new tool called pippin to manage the extlib > > Does this actually exist, or have you picked it because the name is > cool? (It is a cool name.) > Almost everything in the document doesn't exist yet and this in no exception. I think being a Tolkien fan may have something to do with why I picked it. > > (There is already a pippin on PyPI and github. I don't know if they are > the same project. The one of github claims to be experimental. Remember > that most experiments fail.) > > > > * Formalising what a package index is > > What do you mean? > > I can envision a future where there are multiple Python Package Indexes. * The cheeseshop for 'uncontrolled' packages. * A company specific package index that only packages that have been signed by an administrator can be installed. * Plus others Plus indexes in various stages of development * PyPi - The current version * Warehouse - The next version * devpi - a bleeding edge development version If there are to be tools which can interface with all these then there should be a formal document. Does this exist? I could not find one. All I've found various PEPS which propose updates to the cheeseshop. This along with the pypi tool completes the PyPi Trinity. > > > * Addressing a Python project's lifecycle. > > Surely that will be different for every project? > > Sort of, but every project must go through some of these stages. Create, Design, Code, Build, Test, Create Distribution, Upload, Download, Install Consistent configuration between each of these steps would be desirable would it not? Allowing the creation of tools to manage projects. If we don't look at them as a whole then we'll probably end up with inconsistencies. > > > * Configuration handling > > What about it? > > Because it's a thing that could be much more consistent across Python software. What do we store in config files and where do we store them * Sections * Item names * Item values (either singular or in a list) * Item types All configuration file formats can handle these but there's no best practice. Why not formalise it? Which directory should they go in? Again consistency is desirable. Surely we can come up with a 'blessed' set of locations? Then document also attempts to address the layering of configurations across * In Code * Site config * user config * Environment variables * Command line parameters > > > * Removing the use of 'lib' suffixes from almost all packages. > > Why? > > Because it's my vision of a future and I like consistency ;-) The question for me is why is it used in some places in the stdlib and not in others? What purpose does it serve? If it's a good idea shouldn't every package in the stdlib have a lib suffix? About the only place I can see it being useful is importlib because there's already an import statement and you don't want to 'import import' in your code > > > * Claiming any name in PyPi starting with 'py' > > That's going to be *really unpopular*. What is the purpose behind that? > > I'm also sure it would be really unpopular but see the start of my previous response. As it's only a vision of a possible future I'm under no illusion that it's the answer to all Python's 'problems' but it's an attempt to group all the disparate discussions into a single document. My answers may almost all be wrong / unimplementable but the discussions on the mailing list so far seem isolated and this is my attempt to make sense of them all. Regards Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at the-compiler.org Tue Apr 14 17:10:00 2015 From: me at the-compiler.org (Florian Bruhin) Date: Tue, 14 Apr 2015 17:10:00 +0200 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <20150414130104.GF5663@ando.pearwood.info> Message-ID: <20150414151000.GI1788@tonks> * Simon Kennedy [2015-04-14 07:58:43 -0700]: > Why do we have py.exe which finds a Python install but not a virtualenv? It does with 3.5: https://www.python.org/dev/peps/pep-0486/ > > > * Formalising what a package index is > > > > What do you mean? > > > > > I can envision a future where there are multiple Python Package Indexes. > > * The cheeseshop for 'uncontrolled' packages. > * A company specific package index that only packages that have been signed > by an administrator can be installed. > * Plus others > > Plus indexes in various stages of development > > * PyPi - The current version > * Warehouse - The next version > * devpi - a bleeding edge development version > > If there are to be tools which can interface with all these then there > should be a formal document. Does this exist? I could not find one. All > I've found various PEPS which propose updates to the cheeseshop. > > This along with the pypi tool completes the PyPi Trinity. Are you aware of http://doc.devpi.net/latest/ ? This seems to me like the "company-specific package index" you're mentioning here. Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From abarnert at yahoo.com Tue Apr 14 17:43:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 14 Apr 2015 08:43:46 -0700 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <20150414130104.GF5663@ando.pearwood.info> Message-ID: On Apr 14, 2015, at 07:58, Simon Kennedy wrote: > >> On Tuesday, 14 April 2015 14:01:30 UTC+1, Steven D'Aprano wrote: >> >> Well, you've certainly been meticulous :-) >> >> [...] >> > * Removing the notion of 'virtual' environments. Everything's just an >> > environment. >> >> What's the difference? > > Not a lot ;-) Merely a reduction in concepts by 1. > > The question I asked myself when I thought about it is. Why 'virtual'? What's the conceptual difference between a standard Python install and a virtualenv (now that no-site-packages is the default.)? How does it help a Python user to distinguish between the two? Why do we have py.exe which finds a Python install but not a virtualenv? > > So why not merge the Python install and the virtual environment into a single concept - Python Environments > > Then we could have two tools > > 1. 'py' which finds and starts environments. (plus py could automatically find scripts in the bin / scripts / tools directories without having to add them to the path) How exactly does it do that? One of the nice things about virtual environments is that I can put them anywhere--in my home directory, on a network share, on an external drive, etc. I can make them relocatable and deploy them just by copying a directory. And so on. How could a tool possibly find all my environments? More importantly, why would I ever want it to? The reason to have separate environments is so I can switch between them. A tool that treats them all the same as my base install is a tool that defeats the purpose of all of them. And I certainly don't want anything installed in the bin/scripts/tools for any environment on my system to act like it's on my path even though it isn't. > 2. 'pyenv' which manages environments (create, list, show info, activate etc.) > > >> >> >> > * Splitting the standard library in twain. stdlib = 'baselib' + 'extlib'. >> > - baselib = Enough to start up a Python environment / install pip. >> > - extlib = The rest of the current stdlib. Changes more frequently than >> > the baselib. >> >> Hmmm. That would have consequences. Are you okay with abandoning large >> parts of the corporate, government and education sectors? >> >> The thing is, one of the reasons that Python is so successful is that it >> is *batteries included*. That's been a selling point for 20 years. This >> means that using only the standard library, you can be productive and >> get real work done. This is important for people using a standard >> operating environment, or in restricted networks with no access to the >> internet, or in any scenario where people have to get approval before >> they can use a particular software package. Not everyone is able to just >> download arbitrary packages from the Internet and use them on the spur >> of the moment. >> >> With Python, they only need to get *one* package approved: "Can I >> download Python and use it?" It might take six weeks to get approval, >> but once you do, you have not just the core language, but a good 150+ >> solid, useful, powerful libraries (and a few not so good ones). >> >> With your Python 3.9.9, they get the core language, and a mechanism for >> downloading from PyPI, which they are not allowed to do without >> approval, on pain of dismissal. Or not able to do, because the corporate >> network blocks it. And don't even think about sneaking it in via USB key >> from home... So each and every package that needs to be downloaded from >> PyPI could take six weeks to get approval before they can run "pip >> install os". >> >> The thing is, the broader Python ecosystem is not just hackers on Linux >> with control over their own machines. It includes people in schools >> using locked-down SOEs, corporate users with restricted network access >> or paranoid legal/IT departments, people in developing nations with >> unreliable internet access, and more. > > The idea of the baselib + extlib is an attempt to address the desire to update parts of the stdlib faster than every Python release. > > You would still get all the batteries included. When installing a new version of Python the extlib would be installed along with the baselib just not along side the baselib i.e. it would be in a separate on disk structure. > > A new version of Python would be > > * python.exe + baselib > * extlib on disk structure somewhere else So this is basically just PEP 413 in disguise. Have you read the discussions on that PEP, or at least the withdrawal reason? You do add two additional things--moving most of the stdlib into an unspecified different location, and inventing a new tool (but just the name, not anything about what it actually does that couldn't already be done with pip) for doing the updates--but I'm not sure what either of those buys. > The pippin tool which manages the extlib can pull in updates to the on disk repository in a batch as and when the maintainer wants. > and the environment's maintainer can update the environment with the version they want. > > Installation from the extlib disk structure to the current environment is 'just a copy' operation. > > A difference then between a package on PyPi and in the extlib is only that the extlib is 'blessed' by the same people who have control of the Python releases. > > >> >> > * A new tool called pippin to manage the extlib >> >> Does this actually exist, or have you picked it because the name is >> cool? (It is a cool name.) > > Almost everything in the document doesn't exist yet and this in no exception. I think being a Tolkien fan may have something to do with why I picked it. > >> >> (There is already a pippin on PyPI and github. I don't know if they are >> the same project. The one of github claims to be experimental. Remember >> that most experiments fail.) >> >> >> > * Formalising what a package index is >> >> What do you mean? > > I can envision a future where there are multiple Python Package Indexes. > > * The cheeseshop for 'uncontrolled' packages. > * A company specific package index that only packages that have been signed by an administrator can be installed. > * Plus others > > Plus indexes in various stages of development > > * PyPi - The current version > * Warehouse - The next version > * devpi - a bleeding edge development version > > If there are to be tools which can interface with all these then there should be a formal document. Does this exist? I could not find one. All I've found various PEPS which propose updates to the cheeseshop. > > This along with the pypi tool completes the PyPi Trinity. > >> >> > * Addressing a Python project's lifecycle. >> >> Surely that will be different for every project? > > Sort of, but every project must go through some of these stages. > > Create, Design, Code, Build, Test, Create Distribution, Upload, Download, Install Not every project needs all these stages. Not every project cycles through them in the same order. Many projects are developed by organizations that have their own process. Forcing everyone to do it the same way just means stopping many people from doing it at all. > Consistent configuration between each of these steps would be desirable would it not? Like what? Other than the small amount of information that's already in setup.py and the .cnf file, what other configuration do most projects need between any of these steps? > Allowing the creation of tools to manage projects. PyCharm, Eclipse PyDev, etc. already seem to be able to manage projects without anyone doing anything to allow it. What functionality are they missing that you want? > If we don't look at them as a whole then we'll probably end up with inconsistencies. > > >> >> > * Configuration handling >> >> What about it? > > Because it's a thing that could be much more consistent across Python software. > > What do we store in config files and where do we store them > > * Sections > * Item names > * Item values (either singular or in a list) > * Item types > > All configuration file formats can handle these but there's no best practice. No they can't. Plenty of configuration file formats can't handle sections; others handle unlimited levels of hierarchy. Some don't handle types, many handle small fixed sets of types (each different from the fixed set of types another format handles), only a few can handle arbitrary types. And so on. > Why not formalise it? > > Which directory should they go in? Again consistency is desirable. Surely we can come up with a 'blessed' set of locations? Surely not. To the extent that you want that at all, it should be up to the OS, not Python. Consider what happens if Python says "config files on Unix must go in ~/.config", then Apple says "if you put config information anywhere but the defaults database inside your container, your app can't be sold on the App Store". So now nobody can write an app in Python and sell it in the App Store? > Then document also attempts to address the layering of configurations across > > * In Code > * Site config > * user config > * Environment variables > * Command line parameters You almost certainly don't want that to work the same across every Python program. If I'm writing a tool in Python to do something MySQL-related, I want it to handle that stuff the same way as other MySQL-related tools, not the same way as other tools written in Python. >> > * Removing the use of 'lib' suffixes from almost all packages. >> >> Why? > > Because it's my vision of a future and I like consistency ;-) The question for me is why is it used in some places in the stdlib and not in others? What purpose does it serve? If it's a good idea shouldn't every package in the stdlib have a lib suffix? About the only place I can see it being useful is importlib because there's already an import statement and you don't want to 'import import' in your code > >> >> > * Claiming any name in PyPi starting with 'py' >> >> That's going to be *really unpopular*. What is the purpose behind that? > > I'm also sure it would be really unpopular but see the start of my previous response. > > As it's only a vision of a possible future I'm under no illusion that it's the answer to all Python's 'problems' but it's an attempt to group all the disparate discussions into a single document. > My answers may almost all be wrong / unimplementable but the discussions on the mailing list so far seem isolated and this is my attempt to make sense of them all. Why do you think solving separate problems separately ("isolated") is a problem in the first place? Why is discussing a jumble of dozens of barely-formed ideas, which are admittedly almost all wrong / unimplementable, as useful as discussing even one good idea on its own? > > Regards > Simon > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sffjunkie at gmail.com Wed Apr 15 10:26:10 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Wed, 15 Apr 2015 01:26:10 -0700 (PDT) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> Message-ID: On Tuesday, 14 April 2015 14:54:51 UTC+1, Andrew Barnert wrote: > > On Apr 14, 2015, at 05:14, Simon Kennedy > > wrote: > > Call me a fool, but in the vein of the finest of half baked ideas, I > present to you an idea that hasn't even decided on the shape of the tin yet. > > Using a pseudo random selection of ideas set out in various forums > > * python-dev > * distutils-sig > * python-ideas > * import-sig > > I present to you, in the linked PDF, a possible Python future. > > Python 3.9.9 - The 'I Have A Dream' Version > > http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.pdf > http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.xmind > > > Most of this document seems so ambiguous that it's hard to know how to > comment on it. > > Or perhaps 'nebulous' might be better. However it is an 'ongoing thought experiment'. Not everything is fully fleshed out. I have no qualms in admitting that. I'm certainly prepared to scrap the whole lot but currently the discussions are occurring in isolation and this is my attempt to make some sense of them. But I assume because you've commented on it you're not totally uninterested. > For example, you've got a section called "Types" (which I picked > arbitrarily because my PDF reader decided to start with the top right > corner of the giant page 1), which lists the types as "int, bytes, string, > float, list, tuple, named tuple, dict, ordered dict, set, frozenset, enum, > class, module, generator, iterator, iterable, context manager, function". > > What does that mean? > You're right 'Types' was not the correct term. Is there some term for the things that you manipulate in your code as opposed to the things which perform the manipulation? Maybe 'operands'? It was attempt to separate these 2 concepts. > > A good number of these are builtin types in 3.5; the most obvious > implication is that the rest of them should become builtin types in 3.9.9. > But what would it mean for namedtuple to be a type instead of a type > factory? How can "class" be a type? If iterable becomes a concrete type, > does that mean classes that implement the __iter__ method are no longer > iterables, just "iterable-like types" (and likewise for context manager)? > What would be the benefit of renaming "str" to "string"? Does the fact that > you've left out bool and NoneType and method mean those will no longer be > builtin types in 3.9.9? Some of your type names have spaces--major syntax > change to the language, or typo? > p > Or, if this doesn't mean any such thing, and is just an attempt to > categorize the kinds of stuff that "type-related stuff" will have to deal > with in 3.9.9, how is it any different from 3.5, which already has the same > kinds stuff? > > Not everything in the document is different from the way things currently are. It's an attempt to envision what it would all look like at version 3.9.9 > Picking another part of the page at random, you see to have a stuff that > goes into detail about specific paths for Windows vs. FHS/XDG. Does this > mean you want to drop Mac support, remove the ability to configure and > install into /opt/local or ~, etc.? If not, does it mean anything at all? > > Drop OSX? Yes, of course, it's a dying operating system. :-) Again, not everything is complete. But I posted it because I believe the general concepts that I think could be better defined are outlined in the document. It would be hubris on may part if I thought that I would get everything right on the first pass. Parts need to be fleshed out but the concepts are there. > Right above that, how do you expect to automatically work with every > configuration file format, from .ini and .rc files to the Windows registry > (but not Mac defaults...)? Does that mean you're going to provide some > lowest-common-denominator API (a flat list of C-identifier-style string > keys mapping to printable ASCII string values with no newlines seems to be > about as flexible as you can get)? If not, what else could it mean to > support every format? > A lowest common denominator. Yes, a dict (or maybe an OrderedDict). Configuration is for software to use. The aim would be to coalesce all the disparate sources of configuration data into a single structure available to the software. If we have a set of rules that layer the following configuration sources and we end up with a dict that has the sections / items / values how the tool wants them. global, user, project, command line parameters, environment variables, ... The 'layeredconfig' distribution on PyPi does part of what the document proposes but does not appear to concern itself with standard location of config files or a standard mapping between environment variables for example. But what if we defined a mapping between an environment variable name and a section/value in the resulting dict. Or the mapping between an environment variable name and a config file to parse and add to the dict. And what if the location of global / user configuration files was defined then a user of the system would always know where they could find them my ongoing thought experiment on a Python future I would like to see. > Because, is it not written > > If you don't know where you're going, how will you know when you get > there. > > Whether an infinite number of monkeys could shape this into reality may > never be put to the test, but it's a start. > Or it may not be. Or it may be a start and the end. The future's as > tricksy as hobbitses sometimes. > > Highlights include > > * Python The > - Executable > - Language > - Library > - Tools > - Ecosystem > > * Removing the notion of 'virtual' environments. Everything's just an > environment. > > * Splitting the standard library in twain. stdlib = 'baselib' + 'extlib'. > - baselib = Enough to start up a Python environment / install pip. > - extlib = The rest of the current stdlib. Changes more frequently than > the baselib. > > * A new tool called pippin to manage the extlib > > * Formalising what a package index is > > * Addressing a Python project's lifecycle. > > * Configuration handling > > * Removing the use of 'lib' suffixes from almost all packages. > > * Claiming any name in PyPi starting with 'py' > > And more. > > I have aimed for > > * Separation of concerns > * Consistency in naming of tool commands and options. > > Your thoughts are welcome. > > Next steps are > > * Add all relevant PEPs - both proposed and accepted. > * Add all modules to either baselib, extlib or deletelib > * Firm up stdlib / environment / config file locations (system / user, > Windows / Linux / OSX) > * Create outline of pip / twine code to see what can be put into external > packages for re-use / removal from pip. > * Create a filesystem structure for projects and configuration files / > cookiecutter templates to achieve this. > * Enhance definition of tools and propose code structures. > > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Apr 15 13:44:37 2015 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 15 Apr 2015 21:44:37 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> Message-ID: On Wed, Apr 15, 2015 at 6:26 PM, Simon Kennedy wrote: > currently the discussions are occurring in isolation and this is my attempt > to make some sense of them. Based on the subject line, I have to first apologize that I may come off as mean and scary, with a sneer that curdles dairy. But here goes. Feel free to hate me for saying this. :) There is a fundamental difficulty with the unification of the disparate, in that you have to start coping with all sorts of differences and try to make sense of them *in code*. When, for instance, you want to unify config handling, what you're saying is either that every config handler has to devolve to the lowest common denominator, or else that your unified config handler has to become the least common multiple. The latter may seem tempting ("you can use this module to do ANYTHING!"), but you quickly fall foul of the Inner-Platform Effect. http://thedailywtf.com/articles/The_Inner-Platform_Effect https://en.wikipedia.org/wiki/Inner-platform_effect If you want something to be truly customizable, the best way is to make it Turing-complete and then let people write code. CPython is an ultimately-customizable application, and we call its config files "Python scripts". There's not a lot of point adding another infinitely-customizable layer on top of that, just for something as simple as configuration files. Better to allow code to be code. Several of your proposals have merit, but they should be disentangled from the ones which will just drag them down. ChrisA From tjreedy at udel.edu Wed Apr 15 14:28:06 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 15 Apr 2015 08:28:06 -0400 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> Message-ID: On 4/15/2015 4:26 AM, Simon Kennedy wrote: > You're right 'Types' was not the correct term. Is there some term for > the things that you manipulate in your code Everything we manipulate with Python is an object that is an instance of a class. The instance methods of the class of an object are ultimately responsible for all manipulations of the object. > as opposed to the things which perform the manipulation? We manipulate with functions (in the gemeral sense of callable), which are sometime methods and are always objects themselves. This means that functions can also be 'manipulated' (created, called, bound, composed, passed, or returned). > Maybe 'operands'? Operands are arguments of functions 'named' by symbols. --- I appreciate your desire to write up a master plan of things you would like to see done. However, if presented as a package deal, it will be rejected as it includes compatibility breaks that will not happen. The things that are at least somewhat possible will comprise multiple peps. -- Terry Jan Reedy From steve at pearwood.info Wed Apr 15 15:19:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Apr 2015 23:19:44 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> Message-ID: <20150415131943.GJ5663@ando.pearwood.info> On Wed, Apr 15, 2015 at 01:26:10AM -0700, Simon Kennedy wrote: > You're right 'Types' was not the correct term. Is there some term for the > things that you manipulate in your code as opposed to the things which > perform the manipulation? Maybe 'operands'? > > It was attempt to separate these 2 concepts. I believe the distinction you want is "data" versus "code", or perhaps "values" versus "functions". The trouble is that in Python, functions are values too. You can pass a function to a function and manipulate it. The term usually used to describe this is "functions are first-class values". This is a feature, not a problem to be fixed. In my opinion, any programming language which separates "data" from "code" is a language not worth using. If that's not what you mean, you will need to explain in more detail (perhaps with examples?) what you actually mean. -- Steve From sffjunkie at gmail.com Wed Apr 15 15:54:52 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Wed, 15 Apr 2015 06:54:52 -0700 (PDT) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <20150415131943.GJ5663@ando.pearwood.info> References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> <20150415131943.GJ5663@ando.pearwood.info> Message-ID: <9289f828-308f-435c-81fb-3109176da0d3@googlegroups.com> On Wednesday, 15 April 2015 14:22:10 UTC+1, Steven D'Aprano wrote: > > I believe the distinction you want is "data" versus "code", or > perhaps "values" versus "functions". > > The trouble is that in Python, functions are values too. You can pass a > function to a function and manipulate it. The term usually used to > describe this is "functions are first-class values". This is probably what I should have used "First class values" > This is a feature, > not a problem to be fixed. In my opinion, any programming language which > separates "data" from "code" is a language not worth using. > > Oh I agree, it's one the best parts of Python. Not everything in the doc is something that needs to be changed. In fact I think the actual Python language is almost perfect. Simplicity and elegance yet powerful. That part was just an attempt to document the current state of the language (though I admit poorly articulated) > If that's not what you mean, you will need to explain in more detail > (perhaps with examples?) what you actually mean. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rustompmody at gmail.com Wed Apr 15 15:57:43 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Wed, 15 Apr 2015 19:27:43 +0530 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: Message-ID: On Tue, Apr 14, 2015 at 5:44 PM, Simon Kennedy wrote: > Call me a fool, but in the vein of the finest of half baked ideas, I present > to you an idea that hasn't even decided on the shape of the tin yet. > > Using a pseudo random selection of ideas set out in various forums > > * python-dev > * distutils-sig > * python-ideas > * import-sig > > I present to you, in the linked PDF, a possible Python future. > > Python 3.9.9 - The 'I Have A Dream' Version > > http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.pdf > http://www.sffjunkie.co.uk/files/misc/Python%203.9.9.xmind Hi Simon This is a wonderful work. As for the suggestions for '3.9' I hesitate to comment -- I'm really new out here and dont get the sense of what's acceptable or pace of changes. However as a CS-teacher I'm an old geaser [near 30 years] and dont mind saying that an overall ontology for python is a very attractive idea. [Wish I had a less buzzwordy word for ontology...] And one last somewhat OT comment: mindmapping is good and very appropriate for this. But you lose traction by using commercial software. [No I am not a FLOSS bigot but there will be some] Rusi From rosuav at gmail.com Wed Apr 15 16:00:54 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 16 Apr 2015 00:00:54 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <9289f828-308f-435c-81fb-3109176da0d3@googlegroups.com> References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> <20150415131943.GJ5663@ando.pearwood.info> <9289f828-308f-435c-81fb-3109176da0d3@googlegroups.com> Message-ID: On Wed, Apr 15, 2015 at 11:54 PM, Simon Kennedy wrote: > On Wednesday, 15 April 2015 14:22:10 UTC+1, Steven D'Aprano wrote: >> >> I believe the distinction you want is "data" versus "code", or >> perhaps "values" versus "functions". >> >> The trouble is that in Python, functions are values too. You can pass a >> function to a function and manipulate it. The term usually used to >> describe this is "functions are first-class values". > > > This is probably what I should have used "First class values" That's still pretty much everything in Python. Off the top of my head, the only manipulable things that aren't first-class are name bindings themselves, and those can usually be simulated - most namespaces are representable as dictionaries, and other bindings are __[gs]et{item,attr}__ calls; the notable exception is function locals, which you can't easily mutate from outside the function. ChrisA From sffjunkie at gmail.com Wed Apr 15 16:35:23 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Wed, 15 Apr 2015 07:35:23 -0700 (PDT) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> Message-ID: On Wednesday, 15 April 2015 13:28:49 UTC+1, Terry Reedy wrote: > > I appreciate your desire to write up a master plan of things you would > like to see done. If the initial post reads like I think what I've written is perfect and should be adopted wholesale then I apologise. Perhaps my 40+ years should have dampened by belief that if you just aim for a little bit better you might get there but if you try for the improbable you might get a fair bit further. I also thought that the version number of 3.9.9 would be enough of a hint that what's outlined is very long range / speculative and may never occur (I could have used 4.0 but that should probably never happen.). But I also hoped it would generate some ideas. I have faith that Guido and the other core devs know much better than me, after all unlike me they do it as a profession, and would only be a proponent for things which actually work. If they were to run with even the slightest iota of one of the ideas I would be happy. > However, if presented as a package deal, it will be > rejected as it includes compatibility breaks that will not happen. Indeed, I understand that things like claiming all tools / packages must start with py is a non starter as there is hell of a lot of code out there already using that prefix but it's a dream I have. And regarding the 'lib' suffix idea, that was prompted by trying to understand what the hell the suffix actually gets you (if you put lib at the end of some why not put it the end of all, abd why not put a 'pystdlib' prefix on just to make sure) and by 'disallowing' it, hopefully it would stop people using it in the future. > The > things that are at least somewhat possible will comprise multiple peps. > > -- > Terry Jan Reedy > But which, if any? Are they really any good? Would anybody want to implement them? I don't know. They feel right to me but they could all be complete ordure. Regards Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From sffjunkie at gmail.com Wed Apr 15 16:39:18 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Wed, 15 Apr 2015 07:39:18 -0700 (PDT) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> <20150415131943.GJ5663@ando.pearwood.info> <9289f828-308f-435c-81fb-3109176da0d3@googlegroups.com> Message-ID: On Wednesday, 15 April 2015 15:08:57 UTC+1, Chris Angelico wrote: > > > This is probably what I should have used "First class values" > > That's still pretty much everything in Python. Except for statements such as 'if' / 'for' or operators. > Off the top of my head, > the only manipulable things that aren't first-class are name bindings > themselves, and those can usually be simulated - most namespaces are > representable as dictionaries, and other bindings are > __[gs]et{item,attr}__ calls; the notable exception is function locals, > which you can't easily mutate from outside the function. > > Why would you want to mutate function locals from outside the function? That's not something I've ever explored. Regards Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From sffjunkie at gmail.com Wed Apr 15 16:44:36 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Wed, 15 Apr 2015 07:44:36 -0700 (PDT) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: Message-ID: On Wednesday, 15 April 2015 14:58:24 UTC+1, Rustom Mody wrote: > > But you lose traction by using commercial software. > [No I am not a FLOSS bigot but there will be some] > > Rusi > > XMind, for the basics, is actually "free as in beer" now and that's the version I use. http://www.xmind.net/download/portable/ Mac / Linux and Windows versions Regards Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Apr 15 16:53:00 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 16 Apr 2015 00:53:00 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> <20150415131943.GJ5663@ando.pearwood.info> <9289f828-308f-435c-81fb-3109176da0d3@googlegroups.com> Message-ID: On Thu, Apr 16, 2015 at 12:39 AM, Simon Kennedy wrote: > On Wednesday, 15 April 2015 15:08:57 UTC+1, Chris Angelico wrote: >> >> > This is probably what I should have used "First class values" >> >> That's still pretty much everything in Python. > > > Except for statements such as 'if' / 'for' or operators. I say below "manipulable". You can't in any way manipulate an 'if' statement; however, you can play around with the compiled byte-code, which is represented either by a bytestring or a code object, depending on your point of view. >> Off the top of my head, >> the only manipulable things that aren't first-class are name bindings >> themselves, and those can usually be simulated - most namespaces are >> representable as dictionaries, and other bindings are >> __[gs]et{item,attr}__ calls; the notable exception is function locals, >> which you can't easily mutate from outside the function. >> > > Why would you want to mutate function locals from outside the function? > That's not something I've ever explored. In C, you can do it with pointers: void swap(int *x, int *y) { int tmp=*x; *x = *y; *y = tmp; } void func() { int foo, bar; swap(&foo, &bar); } In Python, you can pass locals() to allow another function to _read_ your locals, but not change them. ChrisA From rosuav at gmail.com Wed Apr 15 16:41:50 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 16 Apr 2015 00:41:50 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> Message-ID: On Thu, Apr 16, 2015 at 12:35 AM, Simon Kennedy wrote: > But which, if any? Are they really any good? Would anybody want to implement > them? I don't know. They feel right to me but they could all be complete > ordure. If you think they're all good, then start separate python-ideas threads for each of them. Some are simple enough to be responded to as-is (eg the removal of "lib" - simple proposal and simple response, namely that it'd break backward compat for minimal gain), others will need more discussion. But they'll all get some interest, I think :) ChrisA From sffjunkie at gmail.com Wed Apr 15 17:53:15 2015 From: sffjunkie at gmail.com (Simon Kennedy) Date: Wed, 15 Apr 2015 08:53:15 -0700 (PDT) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> Message-ID: <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> On Wednesday, 15 April 2015 12:44:59 UTC+1, Chris Angelico wrote: > > Based on the subject line, I have to first apologize that I may come > off as mean and scary, with a sneer that curdles dairy. But here goes. > Feel free to hate me for saying this. :) > > I have no problem with "harsh but fair". > There is a fundamental difficulty with the unification of the > disparate, in that you have to start coping with all sorts of > differences and try to make sense of them *in code*. When, for > instance, you want to unify config handling, what you're saying is > either that every config handler has to devolve to the lowest common > denominator, or else that your unified config handler has to become > the least common multiple. The stdlib is full of modules which don't attempt everything but instead attempt to do a useful subset and everyone seems to be happy they're included. Why not configuration file handling? I assume we're writing Python code and that eventually all config data needs to end up in memory that Python can access. Pulling numbers out of my posterior, lets say 80% of projects only need nothing more than a text file with a set of sections with name/value pairs under them. That can be parsed into a dict. And 5% want to use Python code. That when it gets exec'ed a dict comes out the other end And 5% want to use a database. We can produce an adapter (say sqlite) that spits out a dict or they can. And 5% want to use an etcd store. That leaves 5% that want to do something funky. They're on their own There aren't that many configuration storage systems in the world. Then you define the order which sources (command line / env var / global config etc.) are used and merge them into a complete configuration Why not 'bless' some code which simplifies the 95% There's a distribution on PyPi called 'layeredconfig' which does much of what I think could be done. A few Python projects I've encountered seem to follow a similar pattern. I'm not sure that I see the demons you appear to see in config file handling. Perhaps you could give a real world example > The latter may seem tempting ("you can use > this module to do ANYTHING!"), but you quickly fall foul of the > Inner-Platform Effect. > > http://thedailywtf.com/articles/The_Inner-Platform_Effect > https://en.wikipedia.org/wiki/Inner-platform_effect > > That's not a term I've encountered before. But perhaps it's part of my problem but I looked at the Wikipedia page and I'm not sure it makes a case that the effect is bad. For example the first is about Firefox's FTP. I use Sublime Text and some of the packages you can install do similar things but I'm not sure I would say they were wrong to wrap git for example. Then it talks about and RDBMS and the entity-attribute-value model but RDF uses a similar structure and is used with RDBMSs so it can't be all bad. Then XML is mentioned. But I'll not touch that subject. Then ChromeOS. Which people seem to like. So it's not affected it too much. The only downside seems to be "they tend to be slower and if poorly coded, less reliable as well.[*citation needed *]". But you could say that about any code. If you want something to be truly customizable, the best way is to > make it Turing-complete and then let people write code. CPython is an > ultimately-customizable application, and we call its config files > "Python scripts". There's not a lot of point adding another > infinitely-customizable layer on top of that, just for something as > simple as configuration files. Better to allow code to be code. > > Several of your proposals have merit, but they should be disentangled > from the ones which will just drag them down. > > ChrisA > Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Apr 15 18:21:56 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Apr 2015 17:21:56 +0100 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> Message-ID: On 15 April 2015 at 16:53, Simon Kennedy wrote: > Why not configuration file handling? That's what configparser is for. Paul From krystiankichewko at gmail.com Wed Apr 15 18:26:50 2015 From: krystiankichewko at gmail.com (Krystian Kichewko) Date: Wed, 15 Apr 2015 17:26:50 +0100 Subject: [Python-ideas] Previous item from iterator. Message-ID: Hi all! I hope I'm posting in correct place. I would like to propose an idea for iterating over an iterator in reverse. prev(iterator[, default]) Built-in function: it should retrieve previous element from an iterator by calling its __prev__() method. Add new, optional method to iterator protocol: __prev__() This method should return previous element from iterator or raise StopIteration exception if the is no previous element. This should be optional, because it would be hard to implement this behaviour in some iterators (e.g. It would be hard to do this for file iterator that returns lines). Other iterators should be modified to include __prev__() (e.g. itertools.count(), list iterator). If an iterator doesn't support prev TypeError should be raise when prev is called. Please let me know what you think about this? If there are no obvious problems I will write a PEP for this. Thanks, Krystian Kichewko ps. I've tried searching for this idea on python-ideas and couldn't find anything. If I missed it and my email is a duplicate, sorry. From toddrjen at gmail.com Wed Apr 15 18:36:59 2015 From: toddrjen at gmail.com (Todd) Date: Wed, 15 Apr 2015 18:36:59 +0200 Subject: [Python-ideas] Previous item from iterator. In-Reply-To: References: Message-ID: On Wed, Apr 15, 2015 at 6:26 PM, Krystian Kichewko < krystiankichewko at gmail.com> wrote: > Hi all! > > I hope I'm posting in correct place. > > I would like to propose an idea for iterating over an iterator in reverse. > > prev(iterator[, default]) > > Built-in function: it should retrieve previous element from an > iterator by calling its __prev__() method. > > > Add new, optional method to iterator protocol: > > __prev__() > > This method should return previous element from iterator or raise > StopIteration exception if the is no previous element. > > > This should be optional, because it would be hard to implement this > behaviour in some iterators (e.g. It would be hard to do this for file > iterator that returns lines). Other iterators should be modified to > include __prev__() (e.g. itertools.count(), list iterator). > > If an iterator doesn't support prev TypeError should be raise when > prev is called. > > Please let me know what you think about this? If there are no obvious > problems I will write a PEP for this. > > Thanks, > Krystian Kichewko > > ps. I've tried searching for this idea on python-ideas and couldn't > find anything. If I missed it and my email is a duplicate, sorry. > What is the use-case for this? If the iterator can go both directions, what is the case where this would be useful but an object with an index, like a list, wouldn't be? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Apr 15 19:28:56 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 16 Apr 2015 03:28:56 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> Message-ID: <20150415172856.GL5663@ando.pearwood.info> On Wed, Apr 15, 2015 at 08:53:15AM -0700, Simon Kennedy wrote: > The stdlib is full of modules which don't attempt everything but instead > attempt to do a useful subset and everyone seems to be happy they're > included. > > Why not configuration file handling? Python already ships with at least four: configparser, for INI files; json, for JSON files; plistlib, for property lists; xml, for XML files; plus pickle and shelve, although they're not human-readable or -writable. Also sqlite and various other database interfaces. Again, not human-readable, but if Firefox can use sqlite for config information, so can Python applications. Plus there are third-party libraries supporting (for example) YAML. And Unix *rc style config files are easy to use too. They're free-form, so it would be hard to write a generic rc config library, but usually simple enough that you can deal with them on an ad hoc basis. Is there an actual concrete problem with config handling that you cannot solve easily with Python today? -- Steve From rosuav at gmail.com Wed Apr 15 21:40:58 2015 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 16 Apr 2015 05:40:58 +1000 Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> References: <4CC873E3-1675-4A7F-AEBC-2E4AF779DF08@yahoo.com> <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> Message-ID: On Thu, Apr 16, 2015 at 1:53 AM, Simon Kennedy wrote: > On Wednesday, 15 April 2015 12:44:59 UTC+1, Chris Angelico wrote: >> There is a fundamental difficulty with the unification of the >> disparate, in that you have to start coping with all sorts of >> differences and try to make sense of them *in code*. When, for >> instance, you want to unify config handling, what you're saying is >> either that every config handler has to devolve to the lowest common >> denominator, or else that your unified config handler has to become >> the least common multiple. > > > The stdlib is full of modules which don't attempt everything but instead > attempt to do a useful subset and everyone seems to be happy they're > included. > > Why not configuration file handling? > > I assume we're writing Python code and that eventually all config data needs > to end up in memory that Python can access. > > Pulling numbers out of my posterior, lets say 80% of projects only need > nothing more than a text file with a set of sections with name/value pairs > under them. That can be parsed into a dict. A dict of dicts, presumably. This is already handled: https://docs.python.org/3/library/configparser.html Note that it has a couple of "See also" references to other perfectly viable config file formats. > And 5% want to use Python code. That when it gets exec'ed a dict comes out > the other end Why a dict? > And 5% want to use a database. We can produce an adapter (say sqlite) that > spits out a dict or they can. Why a dict? > And 5% want to use an etcd store. > That leaves 5% that want to do something funky. They're on their own > > There aren't that many configuration storage systems in the world. > > Then you define the order which sources (command line / env var / global > config etc.) are used and merge them into a complete configuration > > Why not 'bless' some code which simplifies the 95% > > There's a distribution on PyPi called 'layeredconfig' which does much of > what I think could be done. A few Python projects I've encountered seem to > follow a similar pattern. > > I'm not sure that I see the demons you appear to see in config file > handling. Perhaps you could give a real world example There aren't demons - there are just different requirements. Some programs use YAML configs, for which you need to hunt down a third-party package. Some use JSON, some use configparser, some use Python code directly. Not all of them render down to a dict. >> The latter may seem tempting ("you can use >> this module to do ANYTHING!"), but you quickly fall foul of the >> Inner-Platform Effect. >> >> http://thedailywtf.com/articles/The_Inner-Platform_Effect >> https://en.wikipedia.org/wiki/Inner-platform_effect >> > > That's not a term I've encountered before. But perhaps it's part of my > problem but I looked at the Wikipedia page and I'm not sure it makes a case > that the effect is bad. > > For example the first is about Firefox's FTP. I use Sublime Text and some of > the packages you can install do similar things but I'm not sure I would say > they were wrong to wrap git for example. I don't know about *wrapping* git; the inner-platform effect would be if the Sublime package actually *reimplements* git, which is what happened with Firefox's FTP module. > The only downside seems to be "they tend to be slower and if poorly coded, > less reliable as well.[citation needed]". But you could say that about any > code. Look at the article on thedailywtf.com rather than the Wikipedia page, then. Have a think about what it's doing, then decide for yourself if it's worthwhile. Steven cited four config file handlers that already ship with Python. Add in YAML as well, because it's worth adding, and then come up with a way for a Python program to be able to use any of them. How is your uberconfigparser going to handle this? Probably something like this: import config with open("app_name.json") as f: my_config = config.parser(format=config.JSON, source=f) The only effective way to handle everything would be to have a mode-switch that lets you pick a format. And if you're doing that, you may as well just write it like this: import json with open("app_name.json") as f: my_config = json.load(f) The umbrella module provides nothing that can't be done better with actual code. ChrisA From abarnert at yahoo.com Thu Apr 16 00:25:58 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 15 Apr 2015 22:25:58 +0000 (UTC) Subject: [Python-ideas] Previous item from iterator. In-Reply-To: References: Message-ID: <122785352.3918847.1429136758683.JavaMail.yahoo@mail.yahoo.com> On Wednesday, April 15, 2015 9:37 AM, Todd wrote: >On Wed, Apr 15, 2015 at 6:26 PM, Krystian Kichewko wrote: > >Hi all! >> >>I hope I'm posting in correct place. >> >>I would like to propose an idea for iterating over an iterator in reverse. >> >>prev(iterator[, default]) >> >>Built-in function: it should retrieve previous element from an >>iterator by calling its __prev__() method. >> >> >>Add new, optional method to iterator protocol: >> >>__prev__() >> >>This method should return previous element from iterator or raise >>StopIteration exception if the is no previous element. >> >> >>This should be optional, because it would be hard to implement this >>behaviour in some iterators (e.g. It would be hard to do this for file >>iterator that returns lines). Other iterators should be modified to >>include __prev__() (e.g. itertools.count(), list iterator). >> >>If an iterator doesn't support prev TypeError should be raise when >>prev is called. >> >>Please let me know what you think about this? If there are no obvious >>problems I will write a PEP for this. >> >>Thanks, >>Krystian Kichewko >> >>ps. I've tried searching for this idea on python-ideas and couldn't >>find anything. If I missed it and my email is a duplicate, sorry. >> > >What is the use-case for this? If the iterator can go both directions, what is the case where this would be useful but an object with an index, like a list, wouldn't be? http://stupidpythonideas.blogspot.com/2014/07/swift-style-map-and-filter-views.html discusses one use, starting from the idea of making map and friends return views instead of just iterators, and exploring how Swift makes that work. (Note that map can return views that can be iterated exactly as freely as its argument, but filter can't do anything better than bidirectional iteration even if given a randomly-accessible iterable.) My conclusion was that the fundamental changes required to Python's iteration protocol would probably be more harmful than the benefits, but I tried to write it in such a way that someone else can come to their own conclusions and use my evidence? Meanwhile, the input/forward/bidirectional/random-access categories of iteration are fundamental to the design of the Standard Template Library which became the core of C++'s stdlib (and D's). The Java Collections Framework which became the core of Java's collections in 1.2+, the Swift stdlib's collections, and many other languages and libraries are strongly inspired by the STL. So, there's plenty of more general information out there someone could gather and summarize to make the case (that someone presumably being the OP who wants this feature added), from Stepanov's original paper to tutorials on building sequences and generators for Swift. The paradigm examples of bidirectional iteration are linked-list-like and tree-like iteration. In particular, almost all kinds of sorted containers can be iterated in sorted order bidirectionally, but not randomly. There are also fundamental iterator transformations that can only give you a bidirectional iterator, even if given a randomly-accessible one, like filter or merge. (As opposed to map or reverse, which can always give you the same kind of iterator they were given.) There aren't too many fundamental algorithms that require bidirectional iteration but not random-access iteration, or that can be more efficient with bidirectional than with forward, but there are some. Note that sequences aren't the same thing as randomly-accessible iterators: map doesn't return a sequence. Unless you want to change Python to make sequence-like dynamic views, rather than iterators, the core concept behind everything, which brings us back to the Swift design I mentioned at the top. To me, it seems strange adding bidirectional iteration without also adding forward iteration (which in Python terms would probably mean a __peek__ method), and random-access iteration (which could mean __add__ and __subtract__ a la C pointer arithmetic). There are far more algorithms that require, or can be algorithmically more efficient with or significantly simpler to implement with, forward iterators and random-access iterators, as 30-odd years of experience with STL and its descendants have demonstrated. Even more so in a language that doesn't even have linked lists or sorted collections in its stdlib. Finally, Andrei Alexandrescu has two interesting presentations that challenge the design behind the STL and its descendants; IIRC they're called "25 Years With STL" and "Iterators Must Go". His main conclusion is that the iteration _interface_ used by C++ is definitely a problem, but the basic concept may be workable into something useful. C++11's ranges and Swift's views can be seen as different attempts at this, the former probably not radical enough (it's closer to boost:range, which he'd already shown was insufficient) and the latter probably too radical. From abarnert at yahoo.com Thu Apr 16 00:59:46 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 15 Apr 2015 22:59:46 +0000 (UTC) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: References: Message-ID: <2051477386.3962388.1429138786653.JavaMail.yahoo@mail.yahoo.com> On Wednesday, April 15, 2015 1:27 AM, Simon Kennedy wrote: >On Tuesday, 14 April 2015 14:54:51 UTC+1, Andrew Barnert wrote: >On Apr 14, 2015, at 05:14, Simon Kennedy wrote: >> >> >>Call me a fool, but in the vein of the finest of half baked ideas, I present to you an idea that hasn't even decided on the shape of the tin yet. >>> >>> >>>Using a pseudo random selection of ideas set out in various forums >>> >>> >>>* python-dev >>>* distutils-sig >>>* python-ideas >>>* import-sig >>> >>> >>>I present to you, in the linked PDF, a possible Python future. >>> >>> >>> Python 3.9.9 - The 'I Have A Dream' Version >>> >>> >>>http://www.sffjunkie.co.uk/ files/misc/Python%203.9.9.pdf >>>http://www.sffjunkie.co.uk/ files/misc/Python%203.9.9. xmind >> >> >>Most of this document seems so ambiguous that it's hard to know how to comment on it. >> >> > > >Or perhaps 'nebulous' might be better. However it is an 'ongoing thought experiment'. Not everything is fully fleshed out. It goes beyond "not fully fleshed out"; most of your document has no interpretable meaning to anyone who's not you. It's not just that we don't have specifics; even to get the basic concept of what a piece of the document means requires significant effort to dig it out of your head, because it exists nowhere else. For example, how many emails back and forth did it take before anyone understood that your "types" section was trying to catalog the existing ontology of Python's objects, and didn't actually have any proposed changes or any real content (unless "someone should teach me programming language theory, or at least terminology" is a language proposal). Even when you do seem to have a proposal, it's not clear what it is. It took a half-dozen messages back and forth to get to the idea that your config proposal is for storing lowest-common-denominator content in various different config formats, layered (with command-line flags and env variables) in a way which is similar to layeredconfig but in some unspecified way different? And you're still getting replies that aren't relevant to the idea. There still hasn't been any discussion of anything of substance related to the idea. And repeating over and over again "it's not complete" whenever anyone challenges anything or even asks a question doesn't help anything, it just makes it even harder to get the information out of you, and therefore less worth trying. And we're almost not discussing the two most interesting or important ideas in your "dream" (at least I hope the first one wasn't one of your most important ideas), we're discussing two that someone had to pick at random for lack of any better guide to even get started. If you want your ideas to be discussed, people have to know what the ideas are. You need to take that mental map, plus the context that's in your head, and write up something that suggests the changes you envision. Not a PEP for each one, but a 3-sentence paragraph or 10-point outline for each key idea would be more than sufficient for people to have enough idea of what you're talking about to at least ask the right questions. A list of a few dozen possible changes, described in just enough detail that people can be intrigued by some of them and know what followup questions to ask, might be useful. This document, on its own, is not. From steve at pearwood.info Thu Apr 16 01:03:43 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 16 Apr 2015 09:03:43 +1000 Subject: [Python-ideas] Previous item from iterator. In-Reply-To: <122785352.3918847.1429136758683.JavaMail.yahoo@mail.yahoo.com> References: <122785352.3918847.1429136758683.JavaMail.yahoo@mail.yahoo.com> Message-ID: <20150415230342.GM5663@ando.pearwood.info> On Wed, Apr 15, 2015 at 10:25:58PM +0000, Andrew Barnert wrote: > The paradigm examples of bidirectional iteration are linked-list-like > and tree-like iteration. In particular, almost all kinds of sorted > containers can be iterated in sorted order bidirectionally, but not > randomly. In Python terms though, if you think you want a linked-list, you probably actually want a regular list. I'm not so sure about trees, but the lack of any trees in the std lib is probably a hint that a dict makes a good replacement. [...] > There aren't too many fundamental algorithms that require > bidirectional iteration but not random-access iteration, or that can > be more efficient with bidirectional than with forward, but there are > some. Are any of them unable to be efficiently reimplemented in terms of random access lists? What does bidirectional iteration give us that sequences don't? The rest of your post is interesting, but I don't see the relevance to *Python*. Bidirectional iteration isn't being proposed in a vacuum, it has to compete with existing alternatives such as lists and dicts. -- Steve From steve at pearwood.info Thu Apr 16 01:27:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 16 Apr 2015 09:27:44 +1000 Subject: [Python-ideas] Previous item from iterator. In-Reply-To: References: Message-ID: <20150415232743.GN5663@ando.pearwood.info> On Wed, Apr 15, 2015 at 05:26:50PM +0100, Krystian Kichewko wrote: > Hi all! > > I hope I'm posting in correct place. > > I would like to propose an idea for iterating over an iterator in reverse. > > prev(iterator[, default]) > > Built-in function: it should retrieve previous element from an > iterator by calling its __prev__() method. For experimentation purposes, you can easily write your own: _sentinel = object() def prev(it, default=_sentinel): try: return type(it).__prev__(it) except StopIteration: if default is _sentinel: raise else: return _sentinel ought to do it. Then you can write your own iterators, give them a __prev__ method as well as a __next__ method, and experiment to see what advantages it gives you (if any). Can you demonstrate some useful tasks that are easier with this bidirectional iteration than they currently are? -- Steve From cs at zip.com.au Thu Apr 16 02:22:50 2015 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 16 Apr 2015 10:22:50 +1000 Subject: [Python-ideas] Previous item from iterator. In-Reply-To: References: Message-ID: <20150416002250.GA63217@cskk.homeip.net> On 15Apr2015 17:26, Krystian Kichewko wrote: >I would like to propose an idea for iterating over an iterator in reverse. > >prev(iterator[, default]) > >Built-in function: it should retrieve previous element from an >iterator by calling its __prev__() method. > >Add new, optional method to iterator protocol: > >__prev__() > >This method should return previous element from iterator or raise >StopIteration exception if the is no previous element. In addition to the existing responses (working prototype from Steven, various discussions around lists and dicts), my own suggestion would be an itertools wrapper like this: from itertools import rewindable it = original_iterator bi_dir_it = rewindable(it) where "rewindable" returned an iterator that recorded the values issued by an n existing iterator, probably with an optional upper limit. Then all the rewindable object needs to do is stash values in a list as they are returned, and keep an offset into the stash to implement prev and next. i.e. instead of bulking up the iterator protocol, provide a helper wrapper. That way any iterator can be used with your proposal. Cheers, Cameron Simpson It's better, when you're racing with someone you don't know so well, to stick to the inside line - it's easier to avoid the bits. - Barry Sheene, bike GP commentator From abarnert at yahoo.com Thu Apr 16 02:25:43 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 16 Apr 2015 00:25:43 +0000 (UTC) Subject: [Python-ideas] Python 3.9.9 - The 'I Have A Dream' Version In-Reply-To: <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> References: <91fe02be-9cc6-439b-89a1-d8d858977a21@googlegroups.com> Message-ID: <639701189.3981483.1429143943459.JavaMail.yahoo@mail.yahoo.com> To respond to the one concrete proposal anyone's been able to dig out so far: On Wednesday, April 15, 2015 8:53 AM, Simon Kennedy wrote: >On Wednesday, 15 April 2015 12:44:59 UTC+1, Chris Angelico wrote: >>There is a fundamental difficulty with the unification of the >>disparate, in that you have to start coping with all sorts of >>differences and try to make sense of them *in code*. When, for >>instance, you want to unify config handling, what you're saying is >>either that every config handler has to devolve to the lowest common >>denominator, or else that your unified config handler has to become >>the least common multiple. > > >The stdlib is full of modules which don't attempt everything but instead attempt to do a useful subset and everyone seems to be happy they're included. > >Why not configuration file handling? > >I assume we're writing Python code and that eventually all config data needs to end up in memory that Python can access. > >Pulling numbers out of my posterior, lets say 80% of projects only need nothing more than a text file with a set of sections with name/value pairs under them. That can be parsed into a dict. Except that's not actually a lowest-common denominator. Names that aren't in pure printable ASCII don't work in all of your formats. Neither do names with embedded whitespace. In some formats, on some platforms, names are case-insensitive (anyone who's ever tried to port portable apps to Windows and had to deal with their configured "path" variable having the contents of "PATH" instead can tell you how much fun that is). While you suggested that all of these formats can handle types, the paradigmatic one you just presented, the sectioned .ini format, cannot; it can only handle string values, without newlines or nulls, that can be encoded to the system's default encoding (and remember, for a lot of systems, that's still CP1252 or similarly limited). Most importantly, a sectioned .ini file doesn't represent a dict of key-value pairs, it represents a dict of dicts of key-value pairs. But other formats, like environment variables, _do_ represent a single flat dict. The lowest common denominator between those two things is that absolutely nothing can be stored. Of course people have designed "lowest-common-denominator-ish" config formats?see any portability framework like Qt or JUCE, for obvious examples?but they don't actually try to store in all formats, or layer additional functionality on top of some (e.g., using a flat dict with dotted names to represent hierarchy), or partly abandon the idea of human readability (e.g., strings are HTML-encoded, or you can store non-string data but you get a dump in some binary pickle-like format). If you think you can solve the impossible problem that nobody else has, go solve it; just implying that it would be nice if someone solved it, without even indicating that you've recognized its existence, is not a useful proposal. >And 5% want to use Python code. That when it gets exec'ed a dict comes out the other end >And 5% want to use a database. We can produce an adapter (say sqlite) that spits out a dict or they can. >And 5% want to use an etcd store. >That leaves 5% that want to do something funky. They're on their own > >There aren't that many configuration storage systems in the world. > >Then you define the order which sources (command line / env var / global config etc.) are used and merge them into a complete configuration > >Why not 'bless' some code which simplifies the 95% Because not everything belongs in the stdlib. You need to do the same cost-benefit analysis for this library as for any other. >There's a distribution on PyPi called 'layeredconfig' which does much of what I think could be done. A few Python projects I've encountered seem to follow a similar pattern. OK, so is there something fundamentally wrong with layeredconfig? If so, it would be nice to know _what_. If not, why didn't you just say "add layeredconfig or something equivalent" instead of making us drag the same information out of you in more verbose form? From abarnert at yahoo.com Thu Apr 16 02:46:51 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 16 Apr 2015 00:46:51 +0000 (UTC) Subject: [Python-ideas] Previous item from iterator. In-Reply-To: <1307478463.4000305.1429144547870.JavaMail.yahoo@mail.yahoo.com> References: <20150415230342.GM5663@ando.pearwood.info> <1307478463.4000305.1429144547870.JavaMail.yahoo@mail.yahoo.com> Message-ID: <795417566.3981420.1429145211932.JavaMail.yahoo@mail.yahoo.com> On Wednesday, April 15, 2015 4:04 PM, Steven D'Aprano wrote: > > On Wed, Apr 15, 2015 at 10:25:58PM +0000, Andrew Barnert wrote: > >> The paradigm examples of bidirectional iteration are linked-list-like >> and tree-like iteration. In particular, almost all kinds of sorted >> containers can be iterated in sorted order bidirectionally, but not >> randomly. > > In Python terms though, if you think you want a linked-list, you > probably actually want a regular list. I'm not so sure about trees, but > the lack of any trees in the std lib is probably a hint that a dict > makes a good replacement. Well, there _are_ trees in the stdlib, buried inside application-level things like HTML documents. And, if sorted collections were added to the stdlib, they'd probably be trees too. There's also a linked list buried inside OrderedDict. (You could ask why it doesn't just use a list.) That being said, this is part of the reason I didn't think my related idea needed to be added to Python, or even proposed. Linked lists and trees are very important data structures to recursive-decomposition functional programming, but they're not nearly as important as people think to other paradigms. It's not just Python itself that proves this; std::vector (a resizable array, like Python's list) is by far the most used container in C++, and not including std::unordered_map (a hash table, like Python's dict) was the most widely-touted gap in the stdlib until it was finally added. And the Alexandrescu presentations I mentioned also imply that at least some of the reasons the iterator categories are used in code written for C++ and its descendants is that you're kind of forced into it, and with a bette API many of these uses wouldn't arise. > [...] >> There aren't too many fundamental algorithms that require >> bidirectional iteration but not random-access iteration, or that can >> be more efficient with bidirectional than with forward, but there are >> some. > > Are any of them unable to be efficiently reimplemented in terms of > random access lists? What does bidirectional iteration give us that > sequences don't? Any time you use an iterator transformation like filter or partition, what you get is (or could be) bidirectionally iterable, but it's not randomly accessible. And filtering lists is a pretty common thing to do (common enough for filter to be a builtin, and even for an equivalent operation to have syntactic support). Even when you use iterator transformations like map that are conceptually randomly accessible, it's hard to see how they could be actually randomly accessed in today's Python. We could provide __add__ and __subtract__ a la C pointer arithmetic, but a lot of people would probably prefer not to use them when not necessary. Or we could just make views fundamental and largely sideline iterators (which does work pretty well for numpy), but that would be a huge and radical change to Python. Or we could provide views and iterators side by side, but that makes the language twice as big and twice as hard to fit in your head. > The rest of your post is interesting, but I don't see the relevance to > *Python*. Bidirectional iteration isn't being proposed in a vacuum, it > has to compete with existing alternatives such as lists and dicts. That's kind of my point: bidirectional iteration doesn't fit into Python; forward, bidirectional, and random access iteration _might_ fit into Python, but only by radically changing Python in a way that's almost certainly not acceptable. (In my blog post, at the end, I tried to work out whether there are smaller steps in that direction that might be worthwhile on their own, but none of them seemed worthwhile enough for me to even bring up on python-ideas, much less push for.) From kale at thekunderts.net Thu Apr 16 08:39:37 2015 From: kale at thekunderts.net (Kale Kundert) Date: Wed, 15 Apr 2015 23:39:37 -0700 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: References: Message-ID: <552F5929.4030702@thekunderts.net> Hello everyone, I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be followed by except- and finally-blocks just like try-blocks. For example: with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.') This proposed syntax is semantically equivalent to wrapping a with-block within a try-block like so: try: with open('spam.txt') as file: print(file.read()) finally: print('done.') except IOError: print('No spam here...') I see two advantages to the proposed syntax. First and most obviously, it saves an extra line and an extra indentation level. One line may not be a big deal, but one indentation level can really affect readability. Second and more conceptually, it makes sense to think about exception handling in the context of a with-block. More often than not, if you're using a with-block, you're expecting that something in that block could throw an exception. Usually with-blocks are used to make sure resources (e.g. files, database sessions, mutexes, etc.) are properly closed before the exception is propogated. But I very often want to do some custom clean-up as well (alert the user, etc.). Currently that requires wrapping the whole thing in a try-block, but allowing with-blocks to behave as try-blocks is a more direct way to express what is meant. I was curious how often with-blocks are actually wrapped in try-blocks for no other purpose than catching exceptions raised in the with-block. So I searched through a number of open source projects looking (roughly) for that pattern: Project with [1] try-with [2] ============== ======== ============ django 230 17 ipython 541 8 matplotlib 112 3 moinmoin 10 0 numpy 166 1 pillow/pil 1 0 pypy 254 4 scipy 163 2 sqlalchemy 36 0 twisted 72 1 ============== ======== ============ total 1585 36 (2.27%) [1]: grep -Po '^\s*with .*:' **/*.py [2]: grep -Poz 'try:\s*with .*:' **/*.py Assuming these projects are representative, about 2% of the with-blocks are directly wrapped by try-blocks. That's not a huge proportion, but it clearly shows that this pattern is being used "in the wild". Whether or not it's worth changing the language for the benefit of 2% of with-blocks is something to debate though. What do people think of this idea? -Kale Kundert From abarnert at yahoo.com Thu Apr 16 09:22:22 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 16 Apr 2015 00:22:22 -0700 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <552F5929.4030702@thekunderts.net> References: <552F5929.4030702@thekunderts.net> Message-ID: <526E19F0-152E-4585-B32A-A565821EE004@yahoo.com> On Apr 15, 2015, at 23:39, Kale Kundert wrote: > > Hello everyone, > > I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be > followed by except- and finally-blocks just like try-blocks. For example: > > with open('spam.txt') as file: > print(file.read()) > except IOError: > print('No spam here...') > finally: > print('done.') > > This proposed syntax is semantically equivalent to wrapping a with-block within > a try-block like so: > > try: > with open('spam.txt') as file: > print(file.read()) > finally: > print('done.') > except IOError: > print('No spam here...') Why does the finally come before the except? Also, it's still indented with the with, which means you seem to be defining with-except-finally in terms of with-finally, and not defining with-finally at all. Did you mean to have the finally after the except and dedented? Anyway, I like the idea, but does the grammar work? A with that has an optional except and finally at the same level seems potentially ambiguous. For example: with spam: pass with eggs: pass finally: pass If you think about it, it's pretty obvious that the finally goes with the second with, not the first. But can the parser tell that? Or a human who's scanning the code rather than thinking carefully about it? Or tools like auto-indenters? (I think this problem doesn't arise for try, because it must have at least except or finally.) It should be easy to edit the grammar and build the parser to at least the first part of the question; for the last part, trying to hack up a few different external tools like Emacs python-mode. But for the human question, I'm not sure how to answer it quite so easily... > > I see two advantages to the proposed syntax. First and most obviously, it saves > an extra line and an extra indentation level. One line may not be a big deal, > but one indentation level can really affect readability. Second and more > conceptually, it makes sense to think about exception handling in the context of > a with-block. More often than not, if you're using a with-block, you're > expecting that something in that block could throw an exception. Usually > with-blocks are used to make sure resources (e.g. files, database sessions, > mutexes, etc.) are properly closed before the exception is propogated. But I > very often want to do some custom clean-up as well (alert the user, etc.). > Currently that requires wrapping the whole thing in a try-block, but allowing > with-blocks to behave as try-blocks is a more direct way to express what is meant. > > I was curious how often with-blocks are actually wrapped in try-blocks for no > other purpose than catching exceptions raised in the with-block. So I searched > through a number of open source projects looking (roughly) for that pattern: > > Project with [1] try-with [2] > ============== ======== ============ > django 230 17 > ipython 541 8 > matplotlib 112 3 > moinmoin 10 0 > numpy 166 1 > pillow/pil 1 0 > pypy 254 4 > scipy 163 2 > sqlalchemy 36 0 > twisted 72 1 > ============== ======== ============ > total 1585 36 (2.27%) > > [1]: grep -Po '^\s*with .*:' **/*.py > [2]: grep -Poz 'try:\s*with .*:' **/*.py > > Assuming these projects are representative, about 2% of the with-blocks are > directly wrapped by try-blocks. That's not a huge proportion, but it clearly > shows that this pattern is being used "in the wild". Whether or not it's worth > changing the language for the benefit of 2% of with-blocks is something to > debate though. > > What do people think of this idea? > > -Kale Kundert > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jheiv at jheiv.com Thu Apr 16 09:33:06 2015 From: jheiv at jheiv.com (James Edwards) Date: Thu, 16 Apr 2015 03:33:06 -0400 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <526E19F0-152E-4585-B32A-A565821EE004@yahoo.com> References: <552F5929.4030702@thekunderts.net> <526E19F0-152E-4585-B32A-A565821EE004@yahoo.com> Message-ID: On Thu, Apr 16, 2015 at 3:22 AM, Andrew Barnert wrote: > Anyway, I like the idea, but does the grammar work? A with that has an optional except and finally at the same level seems potentially ambiguous. For example: > > with spam: > pass > with eggs: > pass > finally: > pass Is this any more (or less) ambiguous than: if cond1: pass if cond2: pass else: pass And we have no issue with that (at least not syntactically). With respect to readability however, ... But as far as the example goes, I agree that it's probably a bit mixed up as was stated. From abarnert at yahoo.com Thu Apr 16 09:56:18 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 16 Apr 2015 07:56:18 +0000 (UTC) Subject: [Python-ideas] with-except-finally blocks In-Reply-To: References: Message-ID: <2134642838.4180734.1429170978927.JavaMail.yahoo@mail.yahoo.com> On Thursday, April 16, 2015 12:33 AM, James Edwards wrote: > > On Thu, Apr 16, 2015 at 3:22 AM, Andrew Barnert > wrote: >> Anyway, I like the idea, but does the grammar work? A with that has an > optional except and finally at the same level seems potentially ambiguous. For > example: >> >> with spam: >> pass >> with eggs: >> pass >> finally: >> pass > > Is this any more (or less) ambiguous than: > > if cond1: > pass > if cond2: > pass > else: > pass > > And we have no issue with that (at least not syntactically). Duh. Just to make sure, I hacked up a quick patch. (Diff attached.) This gets you enough to test the syntax: $ ./python.exe >>> import ast >>> print(ast.dump(ast.parse('with:\n pass\nwith:\n pass\nfinally:\n pass\n'))) Module(body=[With(items=[withitem(context_expr=Name(id='spam', ctx=Load()), optional_vars=None)], body=[Pass()], handlers=[], orelse=[], finalbody=[]), With(items=[withitem(context_expr=Name(id='eggs', ctx=Load()), optional_vars=None)], body=[Pass()], handlers=[], orelse=[], finalbody=[]), With(items=[], body=[Pass()], handlers=[], orelse=[], finalbody=[Pass()])]) That's one With statement with no except/else/finally, and one With statement with a finally, just as you'd expect. (Of course the compiler just ignores the except/else/finally bits.) -------------- next part -------------- A non-text attachment was scrubbed... Name: with.diff Type: application/octet-stream Size: 17056 bytes Desc: not available URL: From steve at pearwood.info Thu Apr 16 15:09:13 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 16 Apr 2015 23:09:13 +1000 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <552F5929.4030702@thekunderts.net> References: <552F5929.4030702@thekunderts.net> Message-ID: <20150416130913.GO5663@ando.pearwood.info> On Wed, Apr 15, 2015 at 11:39:37PM -0700, Kale Kundert wrote: > Hello everyone, > > I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be > followed by except- and finally-blocks just like try-blocks. For example: > > with open('spam.txt') as file: > print(file.read()) > except IOError: > print('No spam here...') > finally: > print('done.') This has two obvious, and slightly different, interpretations: with open('spam.txt') as file: try: print(file.read()) except IOError: print('No spam here...') finally: print('done.') try: with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.') I'm not even going to ask you which of the two you intended, because whichever one you pick, half the time people will want the other. And of those times, half the time they will end up using this syntactic sugar anyway, and then be disturbed when it doesn't quite work the way they expected. To put it another way, I expect this to be an oh-so-subtle bug magnet. > This proposed syntax is semantically equivalent to wrapping a with-block within > a try-block like so: > > try: > with open('spam.txt') as file: > print(file.read()) > finally: > print('done.') > except IOError: > print('No spam here...') You can't describe "with...finally" in terms of "with...finally", it gives a RecursionError :-) I assume this is an editing mistake, and you intended the finally clause to follow the except clause. > I see two advantages to the proposed syntax. First and most obviously, it saves > an extra line and an extra indentation level. One line may not be a big deal, > but one indentation level can really affect readability. It certainly can affect readability, but not in the way you mean. The loss of that indentation level *hurts* readability, it doesn't help it, because it obscures the structure of the code. Either the try block encloses the with, or the with encloses the try. They are not part of the same structure in the way that "if...elif... else" or "for...else" are. [Aside: in the second case, the for-loop, "else" is not the best choice of keyword. The semantics are more like "then", but we are probably stuck with it now.] Given either version: with spam: try: X except: ... try: with spam: X except: ... the fact that X is indented two levels is a good thing, not a bad thing. It makes the structure of the code immediately visible, and there is absolutely no question of whether the with is enclosing the try or the try is enclosing the with. Existing Python statements which introduce blocks strictly keep to the rule that things which are semantically at different levels are indented differently. (The exception is the single line form of some statements, e.g. "if condition: pass" on a single line.) We never write things like: if condition: for x in sequence: do_this() try: while flag: do_this() except Error: handle_error() def func(arg): with func(arg) as spam: return spam.thing() even in the cases where they wouldn't be syntactically ambiguous. Such things are strictly disallowed, and that is good. > Second and more > conceptually, it makes sense to think about exception handling in the context of > a with-block. No more than any other block of code. There is nothing special about with blocks that go with exception handling, any more than (say) indexing into a list, processing a dict, or importing a module. > More often than not, if you're using a with-block, you're > expecting that something in that block could throw an exception. "More often than not"? Not according to your own investigation, where you found that only two percent of the time with blocks are associated with try blocks. That means that in the overwhelming majority of cases, nearly 98% of the time, we are NOT catching exceptions in the with-block. That means that *at best* this syntactic sugar will be of little value, and at worst it will be actively harmful. I think it will be actively harmful, and give it a strong -1 vote. -- Steve From abarnert at yahoo.com Thu Apr 16 15:25:37 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 16 Apr 2015 06:25:37 -0700 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <20150416130913.GO5663@ando.pearwood.info> References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> Message-ID: On Apr 16, 2015, at 06:09, Steven D'Aprano wrote: >> Second and more >> conceptually, it makes sense to think about exception handling in the context of >> a with-block. > > No more than any other block of code. There is nothing special about > with blocks that go with exception handling, any more than (say) > indexing into a list, processing a dict, or importing a module. I think the key insight behind this is that a context manager is in some sense equivalent to a try/finally--and can in fact be implemented that way using @contextmanager. That isn't true for any of the other cases. Which also implies that there is "one true answer" to your initial question; the implicit try-equivalent comes after the __enter__. Nevertheless, I see your point, and I think you're right. This interpretation is clearly not obvious and universal (given that you didn't see it that way...). And anyone who _doesn't_ intuitively equate the context manager to the equivalent a try/finally is just as likely to expect either semantics as the other, and therefore it will be a bug magnet and/or a readability issue. (To use it safely, it's not sufficient for _you_ to understand the intended semantics, but for all of your possible readers, after all.) From kale at thekunderts.net Thu Apr 16 17:20:07 2015 From: kale at thekunderts.net (Kale Kundert) Date: Thu, 16 Apr 2015 08:20:07 -0700 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <526E19F0-152E-4585-B32A-A565821EE004@yahoo.com> References: <552F5929.4030702@thekunderts.net> <526E19F0-152E-4585-B32A-A565821EE004@yahoo.com> Message-ID: <552FD327.8030406@thekunderts.net> On 04/16/2015 12:22 AM, Andrew Barnert wrote: > Why does the finally come before the except? Also, it's still indented with the with, which means you seem to be defining with-except-finally in terms of with-finally, and not defining with-finally at all. Did you mean to have the finally after the except and dedented? Oops, my bad. I did mean to put the finally-block after the except-block: try: with open('spam.txt') as file: print(file.read()) except IOError: print('No spam here...') finally: print('done.') From rymg19 at gmail.com Thu Apr 16 17:32:01 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 16 Apr 2015 10:32:01 -0500 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <526E19F0-152E-4585-B32A-A565821EE004@yahoo.com> References: <552F5929.4030702@thekunderts.net> <526E19F0-152E-4585-B32A-A565821EE004@yahoo.com> Message-ID: Is that any more ambiguous than for...else? On Thu, Apr 16, 2015 at 2:22 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Apr 15, 2015, at 23:39, Kale Kundert wrote: > > > > Hello everyone, > > > > I'd like to propose a little bit of syntactic sugar: allowing > with-blocks to be > > followed by except- and finally-blocks just like try-blocks. For > example: > > > > with open('spam.txt') as file: > > print(file.read()) > > except IOError: > > print('No spam here...') > > finally: > > print('done.') > > > > This proposed syntax is semantically equivalent to wrapping a with-block > within > > a try-block like so: > > > > try: > > with open('spam.txt') as file: > > print(file.read()) > > finally: > > print('done.') > > except IOError: > > print('No spam here...') > > Why does the finally come before the except? Also, it's still indented > with the with, which means you seem to be defining with-except-finally in > terms of with-finally, and not defining with-finally at all. Did you mean > to have the finally after the except and dedented? > > Anyway, I like the idea, but does the grammar work? A with that has an > optional except and finally at the same level seems potentially ambiguous. > For example: > > with spam: > pass > with eggs: > pass > finally: > pass > > If you think about it, it's pretty obvious that the finally goes with the > second with, not the first. But can the parser tell that? Or a human who's > scanning the code rather than thinking carefully about it? Or tools like > auto-indenters? > > (I think this problem doesn't arise for try, because it must have at least > except or finally.) > > It should be easy to edit the grammar and build the parser to at least the > first part of the question; for the last part, trying to hack up a few > different external tools like Emacs python-mode. But for the human > question, I'm not sure how to answer it quite so easily... > > > > > > I see two advantages to the proposed syntax. First and most obviously, > it saves > > an extra line and an extra indentation level. One line may not be a big > deal, > > but one indentation level can really affect readability. Second and more > > conceptually, it makes sense to think about exception handling in the > context of > > a with-block. More often than not, if you're using a with-block, you're > > expecting that something in that block could throw an exception. Usually > > with-blocks are used to make sure resources (e.g. files, database > sessions, > > mutexes, etc.) are properly closed before the exception is propogated. > But I > > very often want to do some custom clean-up as well (alert the user, > etc.). > > Currently that requires wrapping the whole thing in a try-block, but > allowing > > with-blocks to behave as try-blocks is a more direct way to express what > is meant. > > > > I was curious how often with-blocks are actually wrapped in try-blocks > for no > > other purpose than catching exceptions raised in the with-block. So I > searched > > through a number of open source projects looking (roughly) for that > pattern: > > > > Project with [1] try-with [2] > > ============== ======== ============ > > django 230 17 > > ipython 541 8 > > matplotlib 112 3 > > moinmoin 10 0 > > numpy 166 1 > > pillow/pil 1 0 > > pypy 254 4 > > scipy 163 2 > > sqlalchemy 36 0 > > twisted 72 1 > > ============== ======== ============ > > total 1585 36 (2.27%) > > > > [1]: grep -Po '^\s*with .*:' **/*.py > > [2]: grep -Poz 'try:\s*with .*:' **/*.py > > > > Assuming these projects are representative, about 2% of the with-blocks > are > > directly wrapped by try-blocks. That's not a huge proportion, but it > clearly > > shows that this pattern is being used "in the wild". Whether or not > it's worth > > changing the language for the benefit of 2% of with-blocks is something > to > > debate though. > > > > What do people think of this idea? > > > > -Kale Kundert > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Apr 16 17:35:14 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 16 Apr 2015 08:35:14 -0700 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <20150416130913.GO5663@ando.pearwood.info> References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> Message-ID: <20150416153514.GA18131@stoneleaf.us> On 04/16, Steven D'Aprano wrote: > On Wed, Apr 15, 2015 at 11:39:37PM -0700, Kale Kundert wrote: >> I'd like to propose a little bit of syntactic sugar: allowing with-blocks to be >> followed by except- and finally-blocks just like try-blocks. For example: -1 'with' is highly specialized, and trying to make it more generic will be confusing. >> Second and more conceptually, it makes sense to think about exception handling >> in the context of a with-block. > > No more than any other block of code. There is nothing special about > with blocks that go with exception handling [...] I have to disagree: 'with' blocks are exactly a condensed try/finally, with the added capability of being able to suppress exceptions -- this makes them very special with respect to exception handling. -- ~Ethan~ From steve at pearwood.info Thu Apr 16 18:25:33 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Apr 2015 02:25:33 +1000 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> Message-ID: <20150416162533.GQ5663@ando.pearwood.info> On Thu, Apr 16, 2015 at 06:25:37AM -0700, Andrew Barnert wrote: > On Apr 16, 2015, at 06:09, Steven D'Aprano wrote: > > >> Second and more > >> conceptually, it makes sense to think about exception handling in the context of > >> a with-block. > > > > No more than any other block of code. There is nothing special about > > with blocks that go with exception handling, any more than (say) > > indexing into a list, processing a dict, or importing a module. > > I think the key insight behind this is that a context manager is in > some sense equivalent to a try/finally--and can in fact be implemented > that way using @contextmanager. That isn't true for any of the other > cases. You are correct that with... statements are, in a sense, equivalent to try...finally. That's hardly a secret or an insight -- I too have read the docs and the PEP :-) But I think it is irrelevant to this question. Anything we write might contain an internal try...finally. Consider: spam() It is, I hope, obvious that there is nothing special about spam that relates it to try...finally more than any other chunk of code. But what if I told you that *inside* spam there was a try...finally clause? def spam(): try: eggs() finally: cheese() I hope that you wouldn't change your mind and decide that spam was special and we should try to create syntactic sugar: spam() finally: tomato() because "spam is in some sense equivalent to a try/finally". Now substitute with statements for spam. Yes, "with..." is fundamentally linked to a finally block. It's part of the definition of the with statement. But that finally block is controlled by the context manager, not us -- as users of the with statement, we don't get to control what happens in the finally clause unless we write our own try...finally block. Exactly the same as spam and its internal finally clause. With the proposed syntax: with something(): block finally: print("Done") there are conceptually two independent finally blocks. There is the finally block hidden inside the context manager's __exit__ method, which is part of the with statement. As a consumer of the CM, we don't have any control over that. And there is our finally block, which prints "Done", which is independent of the context manager and under our control. That raises another interesting question. Which finally clause will trigger first? With current syntax, the block structure is explicit and obvious: with something(): try: block finally: print("Done") "Done" is printed before the __exit__ method is called, because that finally clause is part of the with block. Whereas: try: with something(): block finally: print("Done") in this case, the __exit__ method is called first. The block structure makes that so obvious I don't even need to test it to be sure that it must be the case. (Famous last words!) -- Steven From steve at pearwood.info Thu Apr 16 18:38:57 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Apr 2015 02:38:57 +1000 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <20150416153514.GA18131@stoneleaf.us> References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> <20150416153514.GA18131@stoneleaf.us> Message-ID: <20150416163857.GR5663@ando.pearwood.info> On Thu, Apr 16, 2015 at 08:35:14AM -0700, Ethan Furman wrote: > > No more than any other block of code. There is nothing special about > > with blocks that go with exception handling [...] > > I have to disagree: 'with' blocks are exactly a condensed try/finally, with the > added capability of being able to suppress exceptions -- this makes them very > special with respect to exception handling. Yes, yes, that's a given that with statements encapsulate a finally clause. But that is missing the point. The proposal is not to add a finally clause to context managers, because they already have one. The proposal is to allow special syntax for the situation where you want a separate and independent finally. with spam(): try: blockA except Error: blockB finally: blockC The finally: blockC has nothing to do with the context manager's __exit__ method. We might as well be talking about this: while spam(): try: blockA except Error: blockB finally: blockC giving us syntactic sugar: # not a serious proposal while spam(): blockA except Error: blockB finally: blockC In fact, if I were a betting man, I'd venture a small wager that there are more while- or for-loops with try blocks inside them than with blocks. After all, the whole purpose of with statements is to avoid needing to write a try...finally! -- Steve From python at mrabarnett.plus.com Thu Apr 16 18:50:26 2015 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 16 Apr 2015 17:50:26 +0100 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <20150416163857.GR5663@ando.pearwood.info> References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> <20150416153514.GA18131@stoneleaf.us> <20150416163857.GR5663@ando.pearwood.info> Message-ID: <552FE851.7030203@mrabarnett.plus.com> On 2015-04-16 17:38, Steven D'Aprano wrote: > On Thu, Apr 16, 2015 at 08:35:14AM -0700, Ethan Furman wrote: > >> > No more than any other block of code. There is nothing special about >> > with blocks that go with exception handling [...] >> >> I have to disagree: 'with' blocks are exactly a condensed try/finally, with the >> added capability of being able to suppress exceptions -- this makes them very >> special with respect to exception handling. > > Yes, yes, that's a given that with statements encapsulate a finally > clause. But that is missing the point. The proposal is not to add a > finally clause to context managers, because they already have one. > The proposal is to allow special syntax for the situation where you want > a separate and independent finally. > > with spam(): > try: > blockA > except Error: > blockB > finally: > blockC > > The finally: blockC has nothing to do with the context manager's > __exit__ method. We might as well be talking about this: > > while spam(): > try: > blockA > except Error: > blockB > finally: > blockC > > giving us syntactic sugar: > > # not a serious proposal > while spam(): > blockA > except Error: > blockB > finally: > blockC > If I saw that I'd expect it to mean: try: while spam(): blockA except Error: blockB finally: blockC > > In fact, if I were a betting man, I'd venture a small wager that there > are more while- or for-loops with try blocks inside them than with > blocks. After all, the whole purpose of with statements is to avoid > needing to write a try...finally! > From abarnert at yahoo.com Thu Apr 16 19:29:48 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 16 Apr 2015 10:29:48 -0700 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <20150416162533.GQ5663@ando.pearwood.info> References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> <20150416162533.GQ5663@ando.pearwood.info> Message-ID: <03A8D686-2BBA-427F-A419-C1C500576A8C@yahoo.com> On Apr 16, 2015, at 09:25, Steven D'Aprano wrote: > >> On Thu, Apr 16, 2015 at 06:25:37AM -0700, Andrew Barnert wrote: >> On Apr 16, 2015, at 06:09, Steven D'Aprano wrote: >> >>>> Second and more >>>> conceptually, it makes sense to think about exception handling in the context of >>>> a with-block. >>> >>> No more than any other block of code. There is nothing special about >>> with blocks that go with exception handling, any more than (say) >>> indexing into a list, processing a dict, or importing a module. >> >> I think the key insight behind this is that a context manager is in >> some sense equivalent to a try/finally--and can in fact be implemented >> that way using @contextmanager. That isn't true for any of the other >> cases. > > You are correct that with... statements are, in a sense, equivalent to > try...finally. That's hardly a secret or an insight -- I too have read > the docs and the PEP :-) But I think it is irrelevant to this question. > Anything we write might contain an internal try...finally. Consider: > > spam() > > It is, I hope, obvious that there is nothing special about spam that > relates it to try...finally more than any other chunk of code. But what > if I told you that *inside* spam there was a try...finally clause? > > def spam(): > try: eggs() > finally: cheese() That's clearly different. There is nothing inherent to function call syntax that relates to finally; there is something inherent to a with statement that does. You might just as well argue that there's nothing special about for blocks that go with iteration because any function definition may have a call to iter inside. On top of that, inside a function is an entirely different scope; what happens inside a function is in many senses hidden or isolated from syntax. For example, compare this: for i in range(10): if i == 3: break else: print('broke') for i in range(10): if i == 3: eggs() else: print('broke') If you told me there was a break inside eggs, it wouldn't make any difference. A call to eggs is still not equivalent to a break in this scope, just as a call to spam is still not equivalent to a finally in this scope. > I hope that you wouldn't change your mind and decide that spam was > special and we should try to create syntactic sugar: > > spam() > finally: > tomato() > > because "spam is in some sense equivalent to a try/finally". No more than I'd decide that eggs is special. You could, of course, argue that spam is a different case from eggs because the one exception to scope locality is exceptions, and with statements, unlike break, are fundamentally about exceptions... But that requires assuming exactly the point you're trying to disprove. Again, I agree with you that this proposal is probably a bad idea. But this argument against it doesn't work. From ceridwen.mailing.lists at gmail.com Thu Apr 16 22:22:47 2015 From: ceridwen.mailing.lists at gmail.com (Cara) Date: Thu, 16 Apr 2015 16:22:47 -0400 Subject: [Python-ideas] Pattern matching In-Reply-To: <843638816.3182228.1428623569480.JavaMail.yahoo@mail.yahoo.com> References: <843638816.3182228.1428623569480.JavaMail.yahoo@mail.yahoo.com> Message-ID: <1429215767.9503.22.camel@gmail.com> In my previous post I aimed for brevity and skimmed over some of the details. In some places, I was a bit too brief, and so this post will be much more detailed. > > The second is performing algebra on the struct module's format strings. > > I've written some automatic optimization of them with concatenation and > > scalar multiplication, so that for instance 'b' + 'b' = > > '2b' rather than > > 'bb' and '10s' + '10s' = '10s10s'. The value of > > this is in dealing with > > format strings that can be combined in various ways to describe the > > layouts of complicated binary data formats. With some context stripped > > for brevity, > > > > def _check_add(self, other): > > if not isinstance(other, _StructBase): > > return NotImplemented > > if not self.same_endianness(other): > > return NotImplemented > > if self.endianness == other.endianness or len(self.endianness) > > > len(other.endianness): > > return type(self) > > else: > > if isinstance(other, _StructSequence): > > return type(other) > > else: > > return other._STRUCT_SEQUENCES[other.endianness] > > > > def __add__(self, other): > > struct_sequence = self._check_add(other) > > if isinstance(other, _StructSequence): > > if other._elements == []: > > return struct_sequence(self._elements.copy()) > > else: > > begin = other._elements[0] > > rest = other._elements[1:] > > else: > > begin = other > > rest = [] > > if self._elements == []: > > rest.insert(0, begin) > > return struct_sequence(rest) > > if self._elements[-1].character == begin.character: > > return struct_sequence(self._elements[:-1] + > > [self._elements[-1] + begin] + rest) > > else: > > return struct_sequence(self._elements + [begin] + rest) > > This seems to be overcomplicating things by trying to write ML code in > Python instead of just writing Python code. For example, the only > reason you decompose other._elements into begin and rest is so you can > recompose them (a copy of) into the same list you started with. Why > bother? (Also, if you're avoiding the limited pattern-matching already > available in Python by not writing "begin, *rest = other._elements", > that doesn't make a great case for adding more.) > > def __add__(self, other): > struct_sequence = self._check_add(other) > if isinstance(other, _StructSequence): > elements = other._elements > else: > elements = [other] > if (self._elements and elements > and self.elements[-1].character == elements[0].character): > return struct_sequence(self._elements[:-1] + > [self._elements[-1] + elements[0]] + elements[1:]) > return struct_sequence(self._elements + elements) > > If this is all you're doing, your argument is equivalent to showing > some inherently iterative code written instead in tail-recursive form > to argue that Python should do tail call elimination. You're asking to > make it easier to provide another, less Pythonic way to do something > that can already be easily done Pythonically. I didn't use "begin, *rest = other._elements" because this code has to be Python-2 compatible. (This is another argument for a library solution.) I meant more to give the flavor of the complications of needing to test and decompose objects at the same time in Python than provide the best possible solution. Your version of __add__ is better and simpler than mine, but I still find it hard to read and understand, and you're using _check_add too. The following is another description of this algorithm, simplifying by ignoring the case where you're adding a StructAtom to a StructSequence. (Arguably, that should be another method, or StructAtoms shouldn't be part of the public API just as Python has no char built-in type). A StructSequence is a sequence of StructAtoms. A StructAtom is composed of an endianness, a legal format character, and a positive integer. To concatenate two StructSequences in __add__ or __radd__, here are the steps: 1) Check that `other` is a StructSequence. If it's not, return NotImplemented. 2) Check that `self` and `other` have compatible endiannesses. Probably the best way to do this is use a dictionary with two-element frozensets of endianness characters mapping to one endianness character each. I used conditionals above in _check_add, and that's messier. A missing key means they aren't compatible, and then __radd__ has to raise some kind of exception indicating invalid endiannesses for concatenation. __add__ needs to return NotImplemented. A successful lookup will give the endianness character for the result StructSequence. 3) Check that neither `self` nor `other` are empty. If either is empty, return a copy of the other. (If they're both empty, you'll end up returning a copy of an empty StructSequence, which is fine.) 4) Compare the first StructAtom of one StructSequence with the final StructAtom of the other. (Which is `self` and which is `other` depends on whether we're in __add__ or __radd__.) If the format characters are the same and not 's' or 'p', then create a new StructSequence composed of all but the last character of one, all but the first character of the other, and a StructAtom that has the endianness of the result StructSequence, the same character as the two original StructAtoms, and an integer that's the sum of the two integers in the original StructAtoms. If the format characters aren't the same or are both 's' or 'p', return a new StructSequence that's the concatenation of the two original sequences. Replacing _check_add with the aforesaid dictionary won't improve its clarity much, if any. You still end up with at least four conditionals of some form and needing to do object decomposition in some of the branches. Moreover, struct format string manipulation is simple for problems of these kinds: consider doing simplification on regular expressions. Devin's examples from the ast and sre_compile modules are better illustrations than this. If you don't find the code in these examples hard to read or comprehend, I don't expect you to agree that pattern matching is useful. Do you or don't you? That's an honest question. > > Those are two small-scale examples. Some broad areas where pattern > > matching is useful are: > > > > 1) Parsing: It's useful both for performing manipulations on grammars > > like left factoring and operating on the trees that parsers produce. > > Parsing, and related things (compiling, schema validation, etc.) looks > like a good case, but I'm not sure it is. > > For simple languages, an expression-tree library like PyParsing or > Boost.Spirit seems to be more than sufficient, and often be even nicer > than explicit pattern matching. > > For more complex languages (and sometimes even for simple ones), you > can use or write a framework that lets you write a fully-declarative > syntax and then executes it or generates executable code from it. For > example, if you want to validate and parse an XML language, doing it > by hand in ML would be nicer than in Python, but writing it in XML > Schema and using a schema-validating parser is much nicer than either. > Parsing the Python grammar by hand would be nicer in ML than in C or > Python, but writing it as a declarative GRAMMAR file is nicer than > either. And so on. You could argue that the work to write that > framework is wasted if it's only going to be used once, but how many > complex languages really need to be parsed only once?Even CPython's > parser, which isn't used for anything but parsing Python's GRAMMAR, is > effectively reused every time Python's syntax changes, and every time > someone wants to hack on it to play with new syntax. > > The one really good case for pattern matching seems to be when you > pretty much _have_ to write a custom parser--maybe because the > language is full of special-case rules and tangled context, like C++. > At least the GCC and LLVM/Clang groups gave up trying to write C++ as > a declarative specification and instead wrote custom recursive-descent > parsers in C, which I suspect could be a lot nicer with pattern > matching. But has anyone written an equivalent C++ parser in Haskell, > ML, etc. to compare, or is that just a suspicion? I became interested in pattern matching when I started writing a parsing library and realized that both the manipulations I needed to perform in the parser itself and the manipulations I would need to do on the AST after parsing were better expressed using pattern matching than the if-elif-else trees I was writing. As part of this, I've been doing some informal research on how people do parsing in the real world. Unfortunately, I don't have any formal data and don't know how I'd collect it, not to mention not having the money or time to do a formal survey even if I knew how to do it. With that caveat, some patterns I've noticed: 1) Hand-written recursive-descent parsers are ubiquitous in real code. If you take a random program that needs to do parsing somewhere, I'd give better than even odds that it's done with a hand-written recursive-descent parser, not a library. I have some guesses as to why this is, but I won't speculate here. I think there's a solid case for putting a parser package in the standard library---Haskell and Scala, for instance, have already done this---which helps with one part of this problem, but then you have to do something with the tree you've parsed, and pattern matching is also useful for that. 2) Many programmers don't know (E)BNF or formal grammars. I prefer to use some declarative DSL too, but I know the languages and the concepts already. Those who don't might benefit from features that make parsers easier to write without that background. 3) The world is filled with obscure data formats that none of us have ever heard of; this is is especially true of binary data formats. Each of these requires some new parser whether hand-written, built with a library, or generated. 4) Most languages and formats can't be parsed with anything less than a context-sensitive grammar, and sometimes require more than that. To handle this problem, successful parser libraries make it easy to add custom code to the parsing process. Thus, there's a lot of parsing code in the world, whether there should be or not, and even if you know how formal grammars and how to use parser generators and similar tools, you might still end needing to write parsing code because you have an obscure format to parse, a format that has non-standard properties, or both. Aside: if you do end up needing to write a framework for a language or format, wouldn't pattern matching make it easier to write that framework? As far as I know, there's no C++ parser in Haskell or ML, mainly because C++ is so hard to parse. There exist parsers for other languages (C, for instance) that do make extensive use of pattern matching. > > 2) Mathematics: manipulations of abstract mathematical objects like > > algebraic expressions (not numerical analysis), for instance algebraic > > simplification rules. > > Are you talking about parsing the expressions into a tree? If so, we're still on parsing. I'm talking about doing manipulations on mathematical objects, like say doing substitutions with trigonometric identities. > > 3) Recursive data structures: Python doesn't have algebraic data types > > or embed recursion into everything the way languages like Haskell do, > > but there's plenty of recursively-structured data out in the wild like > > HTML and JSON. > > The parsing of this data is covered above. But what do you do with it > once it's parsed? If you're transforming it, that again seems to be a > case where writing the transformer declaratively in, say, XSSL wins. > And if you're just extracting data from it, usually you're better off > with the simple data['machines'][0]['name'] syntax (in an except > KeyError block), and when that's not good enough, you're usually > better off with a declarative search language like XPath. So this > seems to be the same as with parsing. What is XSSL? All my comments above on parsing apply here, too. One problem I've encountered is that since most languages and formats don't have nice solutions like XPath already available, you end up implementing your own. > > > > Whether I've convinced you or not that pattern matching is a worthwhile > > addition to the language, I'd like it, and if I could, I'd write a > > library to get it. > > That raises the question: what's wrong with the more than a dozen libraries already written? The short answer is some combination of poor performance, lack of compatibility (with 2.7, with 3.4, with PyPy, with other alternate interpreters), bad syntax, and missing functionality, depending on the library. If you have a particular library you're curious about, I can be more specific. > > with match(expr) as case: > > with case(pattern) as matches: > > if matches: > > target_list = matches > > suite > > with case(pattern) as matches: > > if matches: > > target_list = matches > > suite > > ... > > > First, this seems like it will execute _every_ branch that matches, > instead of just the first one. Surely you don't want that, do you? You > can wrap the whole thing in a "while True:" and have each suite (and > the "else" case) end with "break", or wrap it in a try/except and have > each suite raise a custom exception, but you need some way to get out. Yes, I have `while True:` in my notes, it was a clumsy copy-paste, and omitting the break lines was a simple mistake. There are some reasons that try/except might be better. > Also, why not just put the target_list in the as clause (inside > parens, to fit it in a target, which is what an as clause actually > takes)? Of course that means you get an exception rather than an else > when it doesn't match, but that's fine. (It would be nice if it were a > subclass of ValueError, in case the suite raises some ValueError of > its own, however...) I didn't know you could do that :). Yes, that's preferable. > > > > for target_list in pattern(expr): > > suite > > > What is this going to do? From your above example, if looks like > match(expr)(pattern) returns either an iterable that can be assigned > to a target_list or None. What does pattern(expr) return? Return an > iterable of one target_list or an empty iterable? > > Also, the very idea of having an object which can be called on an > expression or can have a (processed) expression called on it with > similar but not identical results seems very confusing. Here, what should happen is that `expr` should be an iterator and calling `pattern()` on it returns another iterator containing the deconstructed parts of each object in the original iterator *if* they match. I agree the lack of parallelism is confusing. > > The useful part of pattern matching--and the hard part, too--is > decomposing structures. A proposal that ignores that and only > decomposes lists and dicts misses the point. > > Yes, it's true that Python doesn't have ADTs. So that means one of two > things: (1) types that want to be matched will have to do a bit of > work to expose their structure and/or to decompose themselves, or (2) > pattern matching isn't useful in Python. The examples you give from > other languages show that the first is at least plausible. And both > Szymon's proposal and mine and various other ideas suggested on this > thread show attempts at a Pythonic design. But just ignoring structure > decomposition, as you're doing, is not a way toward anything useful. I assumed the answer to this was obvious because you've already given it :). Objects have to opt in to pattern matching by having some method that inverts their constructor or at least provides some plausible facsimile thereof. For any object-oriented language, if you want to preserve the ability to define with any attributes and have pattern matching, you have to do it this way or with structural typing, and Python doesn't really have that. Python classes don't have private variables like Java except by convention, but neither do they define their variables except by convention. __init__ is a black box that could be doing any Turing-complete computation on its arguments and with properties, any attribute access could be the result of a computation rather than a member of the class. Scala calls its pattern-matching method unapply by analogy with its apply method. In Python, I'd probably pick a different name, but the concept is the same. > If you're willing to use MacroPy, that brings up the question: what's > wrong with the pattern matching that it comes with as an example, and > why don't you want to build on it instead of starting over? I'm tempted to answer, "Nothing!" because I am thinking about building on it, but the short answer is that I'm not sure the syntactic and performance advantages of macros outweigh the usability problems introduced by macros (unfamiliarity of macros to most Python developers, bootstrapping, complication of the implementation). This is my major concern: if you have a case to make one way or the other, I'd like to hear it. Other secondary concerns are that, despite doing AST rewriting, MacroPy's pattern matching doesn't seem to do any optimization, it isn't feature-complete (it's still listed as experimental for good reasons), and MacroPy itself requires updating to 3.4 (and soon 3.5). None of these are insurmountable problems if I were convinced that macros were the right approach. I would prefer to build off one of the existing solutions rather than make a new one. Unfortunately, for most of them it would be more work to comprehend the existing code than it would be to reimplement their functionality, especially for the ones with nonexistent documentation. In that respect, MacroPy's is definitely one of the more promising options. > > > > I haven't talked at all about the nitty-gritty of implementation, though > > I can. I think the most productive way to have that discussion on this > > list would be to look at what smaller features would be most helpful for > > implementing pattern matching as a library, and likewise for syntax, > > shades of the recent discussion about macros. > > This last part, I think, is very useful: After building the best > pattern-matching library you can with Python today, figure out what > small changes to the language would allow you to either make it better > or simplify its implementation, then propose those small changes > instead of proposing pattern matching itself. For example, if you have > to use nonportable hacks (or a MacroPy macro) so your context > manager's __enter__ method can skip the statement body, that would be > a good argument for reopening PEP 377 (because you have a new use case > that wasn't known at the time). But trying to guess in advance what > features might be helpful seems harder and less likely to pay off. > > That being said, I suspect it's not going to pay off, because I think > the with statement is a false lead in the first place. The semantics > look promising, but they're not actually what you want to build on > (unless you want to completely transform with statements into > something that isn't actually a with statement, using MacroPy AST > transformations or otherwise). I'd love to be proven wrong, of course. After some thinking about it, I've concluded you're right, because both of the other options (try and for) combine binding with control flow, while with requires an additional control flow statement without something like PEP 377. Let's leave aside MacroPy and other ways of changing Python's semantics like the decorators in Grant's PyPatt and \url{https://github.com/Suor/patterns} for the moment. Pattern matching has two basic forms, the case/match block and function definition, but before I can talk about them, I have to discuss patterns themselves. Without metaprogramming, patterns have to be defined either as ordinary functions or classes. You have some set of elementary patterns like class patterns, dict patterns, list patterns, variable patterns, and so on. You construct or initialize patterns with either objects to match against or other patterns, mimicking the structure of the data you want to match. With the same tentative syntax I suggested last time: ClassPattern(Foo(1, VariablePattern('a', str))) DictPattern['a': 1, 'b': 2, 'c': 'a': int] ListPattern[1, ..., 'a': Bar] Alternately, if you don't want to use the slicing syntax like I am there, you can make everything very explicit: ClassPattern(Foo(1, VariablePattern('a', str)) DictPattern(('a', 1), ('b', 2), ('c', VariablePattern('a', int))) ListPattern(1, ..., VariablePattern('a', Bar)) Even the slicing syntax is verbose. Both forms of syntax could be made less verbose. Would that make it more or less clear? I'm not sure. This syntax is tightly circumscribed because the whole point of pattern matching is that the patterns should look like the objects they're being matched against. The closer the resemblance, the happier I'd be. After defining a pattern, you call it or one of its methods on the object you want to match against, and it returns either some indication of failure, maybe None, maybe some object with information about the non-match, or an object containing the deconstructed objects in a form suitable for tuple assignment. The case/match block should start with the object that the patterns are to be matched against (avoiding this duplication is a syntactic advantage of pattern matching over if/elif/else blocks), contain a list of patterns to match against, have access to variables in the local scope, and bind variables in the same scope. The only statements that bind variables in the same scope beyond basic assignment statements in Python are, as far as I know, try-except, with, and for. (Well, import, but it's obviously unsuitable.) Here are some sketches of syntax using try-except and for. try: pattern0(expression, pattern1, pattern2, ...) except pattern0.exception as identifier: target_list = identifier suite except pattern1.exception as identifier: target_list = identifier suite except pattern2.exception as identifier: target_list = identifier suite ... else: suite Here, each pattern has an associated exception. If a pattern matches, it throws its exception; if it doesn't match, it calls the first pattern in its varargs list on the value of the expression it was passed, and so on. The last pattern has no pattern to call so returns None if it fails to match. The exceptions have iterator methods for tuple unpacking their arguments after they've been caught. Another possible choice for setting up the patterns uses operator overloading and a dummy function to look vaguely like ML's syntax: match(expression, (pattern0 | pattern1 | pattern2 ...)) Using for statements can avoid the extra assignment statements at the cost of needing to reiterate the expression and some raises. try: for target_list in pattern0(expression): suite raise MatchFound for target_list in pattern1(expression): suite raise MatchFound ... except MatchFound: pass else: suite A pattern either returns a one-element iterator containing an iterator to be bound to target_list or raises StopIteration, in which case the loop body gets skipped and the next pattern is tried. Once any loop body is executed, it raises MatchFound, which skips all the other patterns. In Python, function definition is easier both for syntax and implementation. All the pattern matching implementations for Python that dispatch to functions ( http://blog.chadselph.com/adding-functional-style-pattern-matching-to-python.html , https://github.com/martinblech/pyfpm , http://svn.colorstudy.com/home/ianb/recipes/patmatch.py , https://github.com/admk/patmat , http://peak.telecommunity.com/DevCenter/PEAK-Rules ) use decorators and all use them in similar ways. There are two major variations. First, some decorators take patterns as arguments while others use introspection on keyword arguments to embed the patterns in the function definition. You could use annotations instead of keyword arguments, though I don't know how easily that would integrate with type-hinting---it's probably possible because pattern matching and typing are closely related. I assume the reason no one's done so is because annotations are Python-3 only. Second, some implementations have one uniform decorator across all definitions of a function with frame-hacking or similar hacks to introspect the namespace and link up the decorated functions, while others use two decorators like functools.singledispatch, one to create a pattern-matched function and another to link other functions to the first one. I prefer the latter approach because I dislike frame-hacking on principle, since it introduces compatibility issues, and I think two decorators is more explicit as to what's going on. Beyond those differences, they're all similar: there's an outer function that tries patterns against its arguments until it finds one that matches, then it takes the names bound by the pattern and passes them as arguments to the inner function. Quoting my suggested function definition syntax from my last message: @match(pattern) def f(parameter_list): suite @f.case(pattern) def g(parameter_list): suite @f.case(pattern) def h(parameter_list): suite f(expression) In all of these potential syntax choices, the names of bound variables are assigned manually in assignment statements or function arguments. These approaches all have some boilerplate, but I'm more worried about overhead. Without any desugaring, most of them will require at least one function call per pattern on top of the conditionals and copying necessary. With metaprogramming, it's possible to eliminate some of the function calls for function definitions and the pseudo-ML match-case statement by constructing a single dispatch function or single pattern before calling any patterns. Unfortunately, in the match-case statement, moving overhead to pattern definition time doesn't help because the patterns have to be redefined every time they're executed unless they're explicitly defined outside it, making the code less readable; this is another way in which pattern matching for function definition is easier. Any metaprogramming could remove the boilerplate, but MacroPy could move the overhead to module import time. This is, I think, the best reason to use MacroPy over both non-metaprogramming approaches and other metaprogramming approaches. Unfortunately, Python's current implementation imposes some limitations on desugaring. In ML and Haskell, desugaring pattern matching involves creating a finite automaton. In Python, the VM doesn't provide any way to jump to an offset determined by runtime information short of Philip J. Eby's computed goto hack ( https://mail.python.org/pipermail/python-dev/2006-June/066115.html ), which isn't portable. This means that any state with more than two outgoing transitions has to be represented with an if-else tree that turns what could be a constant-time operation (look up the jump location in a hash table, jump) to a linear-time operation (check conditionals equal to one minus the number of transitions). This is, of course, the switch statement, PEPs 275 and 3103, in another guise. Another, less serious problem is there's a constant-time performance penalty from needing to use exceptions to implement backtracking or nodes with more than one incoming transition. Supposing I were to build on MacroPy's pattern matching, is MacroPy's current syntax for the match-case statement good enough? If not, how could it be improved? As I understand it, by using import hooks like MacroPy does and a new parser, the sky's the limit as far as changing syntax goes, but I'd much rather stay within MacroPy's limitations, using Python's current parsers, than go outside them. > Meanwhile transforming a pattern-match into something that can be > assigned to a target_list instead of just assigning in the match does > work in simple cases, but I think Szymon's proposal shows why it's not > sufficient, and I think any real examples will show that it's > redundant. Even without the language support in my proposal, explicit > quoting of names like in Grant Jenks' PyPatt library shows that you > can get most of the way there. (The "as" clause is useful for the case > when you want to bind the whole structure at the same time you > decompose it and potentially bind its parts, but that's pretty > rare--which is another reason I think "with" is misleading.) I don't understand your comment about how Szymon's proposal shows assigning to a target_list isn't sufficient. Cara From ethan at stoneleaf.us Thu Apr 16 23:46:49 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 16 Apr 2015 14:46:49 -0700 Subject: [Python-ideas] with-except-finally blocks In-Reply-To: <20150416163857.GR5663@ando.pearwood.info> References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> <20150416153514.GA18131@stoneleaf.us> <20150416163857.GR5663@ando.pearwood.info> Message-ID: <20150416214649.GA28035@stoneleaf.us> On 04/17, Steven D'Aprano wrote: > On Thu, Apr 16, 2015 at 08:35:14AM -0700, Ethan Furman wrote: >> Steven D'Aprano wrote: >>> >>> No more than any other block of code. There is nothing special about >>> with blocks that go with exception handling [...] >> >> I have to disagree: 'with' blocks are exactly a condensed try/finally, with the >> added capability of being able to suppress exceptions -- this makes them very >> special with respect to exception handling. > > Yes, yes, that's a given that with statements encapsulate a finally > clause. But that is missing the point. No, that's entirely the point. You claimed: >>> There is nothing special about with blocks that go with exception handling when in fact exception handling (even if just ignoring) is half their reason for existence. It was an unfortunate choice of words, not a big deal. Nobody who has called you on it has suggested the proposal is a win for Python syntax. -- ~Ethan~ From daoust.mj at gmail.com Fri Apr 17 12:15:43 2015 From: daoust.mj at gmail.com (Mark Daoust) Date: Fri, 17 Apr 2015 06:15:43 -0400 Subject: [Python-ideas] Previous item from iterator. Message-ID: If you have python code that really would benefit from iterating in reverse, there already is the "__reversed__" special method, and "reversed" builtin. The doc string for it says "Return a reverse iterator" You guys probably already know about it. It just hadn't been mentioned, so I thought I'd put it out there. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Fri Apr 17 20:58:58 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 14:58:58 -0400 Subject: [Python-ideas] async/await in Python Message-ID: <553157F2.8060408@gmail.com> Hello python-ideas, Here's my proposal to add async/await in Python. I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, and I hope that this PEP can be another big step. Who knows, maybe it will be one of the reasons to drive people's interest towards Python 3. PEP: XXX Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov Discussions-To: Python-Dev Python-Version: 3.5 Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Post-History: Resolution: Abstract ======== This PEP introduces new syntax for coroutines, asynchronous ``with`` statements and ``for`` loops. The main motivation behind this proposal is to streamline writing and maintaining asynchronous code, as well as to simplify previously hard to implement code patterns. Rationale and Goals =================== Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * it is easy to confuse coroutines with regular generators, since they share the same syntax; async libraries often attempt to alleviate this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); * it is not possible to natively define a coroutine which has no ``yield`` or ``yield from`` statements, again requiring the use of decorators to fix potential refactoring issues; * support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification ============= This proposal introduces new syntax and semantics to enhance coroutine support in Python, it does not change the internal implementation of coroutines, which are still based on generators. It is strongly suggested that the reader understands how coroutines are implemented in Python (PEP 342 and PEP 380). It is also recommended to read PEP 3156 (asyncio framework). From this point in this document we use the word *coroutine* to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based on generator syntax. New Coroutine Declaration Syntax -------------------------------- The following new syntax is used to declare a coroutine:: async def read_data(db): pass Key properties of coroutines: * Coroutines are always generators, even if they do not contain ``await`` expressions. * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. * Internally, a new code object flag - ``CO_ASYNC`` - is introduced to enable runtime detection of coroutines (and migrating existing code). All coroutines have both ``CO_ASYNC`` and ``CO_GENERATOR`` flags set. * Regular generators, when called, return a *generator object*; similarly, coroutines return a *coroutine object*. * ``StopIteration`` exceptions are not propagated out of coroutines, and are replaced with a ``RuntimeError``. For regular generators such behavior requires a future import (see PEP 479). types.async_def() ----------------- A new function ``async_def(gen)`` is added to the ``types`` module. It applies ``CO_ASYNC`` flag to the passed generator's code object, so that it returns a *coroutine object* when called. This feature enables an easy upgrade path for existing libraries. Await Expression ---------------- The following new ``await`` expression is used to obtain a result of coroutine execution:: async def read_data(db): data = await db.fetch('SELECT ...') ... ``await``, similarly to ``yield from``, suspends execution of ``read_data`` coroutine until ``db.fetch`` *awaitable* completes and returns the result data. It uses the ``yield from`` implementation with an extra step of validating its argument. ``await`` only accepts an *awaitable*, which can be one of: * A *coroutine object* returned from a coroutine or a generator decorated with ``types.async_def()``. * An object with an ``__await__`` method returning an iterator. Any ``yield from`` chain of calls ends with a ``yield``. This is a fundamental mechanism of how *Futures* are implemented. Since, internally, coroutines are a special kind of generators, every ``await`` is suspended by a ``yield`` somewhere down the chain of ``await`` calls (please refer to PEP 3156 for a detailed explanation.) To enable this behavior for coroutines, a new magic method called ``__await__`` is added. In asyncio, for instance, to enable Future objects in ``await`` statements, the only change is to add ``__await__ = __iter__`` line to ``asyncio.Future`` class. Objects with ``__await__`` method are called *Future-like* objects in the rest of this PEP. Also, please note that ``__aiter__`` method (see its definition below) cannot be used for this purpose. It is a different protocol, and would be like using ``__iter__`` instead of ``__call__`` for regular callables. It is a ``SyntaxError`` to use ``await`` outside of a coroutine. Asynchronous Context Managers and "async with" ---------------------------------------------- An *asynchronous context manager* is a context manager that is able to suspend execution in its *enter* and *exit* methods. To make this possible, a new protocol for asynchronous context managers is proposed. Two new magic methods are added: ``__aenter__`` and ``__aexit__``. Both must return an *awaitable*. An example of an asynchronous context manager:: class AsyncContextManager: async def __aenter__(self): await log('entering context') async def __aexit__(self, exc_type, exc, tb): await log('exiting context') New Syntax '''''''''' A new statement for asynchronous context managers is proposed:: async with EXPR as VAR: BLOCK which is semantically equivalent to:: mgr = (EXPR) aexit = type(mgr).__aexit__ aenter = type(mgr).__aenter__(mgr) exc = True try: try: VAR = await aenter BLOCK except: exc = False exit_res = await aexit(mgr, *sys.exc_info()) if not exit_res: raise finally: if exc: await aexit(mgr, None, None, None) As with regular ``with`` statements, it is possible to specify multiple context managers in a single ``async with`` statement. It is an error to pass a regular context manager without ``__aenter__`` and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use ``async with`` outside of a coroutine. Example ''''''' With asynchronous context managers it is easy to implement proper database transaction managers for coroutines:: async def commit(session, data): ... async with session.transaction(): ... await session.update(data) ... Code that needs locking also looks lighter:: async with lock: ... instead of:: with (yield from lock): ... Asynchronous Iterators and "async for" -------------------------------------- An *asynchronous iterable* is able to call asynchronous code in its *iter* implementation, and *asynchronous iterator* can call asynchronous code in its *next* method. To support asynchronous iteration: 1. An object must implement an ``__aiter__`` method returning an *awaitable* resulting in an *asynchronous iterator object*. 2. An *asynchronous iterator object* must implement an ``__anext__`` method returning an *awaitable*. 3. To stop iteration```__anext__`` must raise a ``StopAsyncIteration`` exception. An example of asynchronous iterable:: class AsyncIterable: async def __aiter__(self): return self async def __anext__(self): data = await self.fetch_data() if data: return data else: raise StopAsyncIteration async def fetch_data(self): ... New Syntax '''''''''' A new statement for iterating through asynchronous iterators is proposed:: async for TARGET in ITER: BLOCK else: BLOCK2 which is semantically equivalent to:: iter = (ITER) iter = await type(iter).__aiter__(iter) running = True while running: try: TARGET = await type(iter).__anext__(iter) except StopAsyncIteration: running = False else: BLOCK else: BLOCK2 It is an error to pass a regular iterable without ``__aiter__`` method to ``async for``. It is a ``SyntaxError`` to use ``async for`` outside of a coroutine. As for with regular ``for`` statement, ``async for`` has an optional ``else`` clause. Example 1 ''''''''' With asynchronous iteration protocol it is possible to asynchronously buffer data during iteration:: async for data in cursor: ... Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows of data from a database after every ``N`` iterations. The following code illustrates new asynchronous iteration protocol:: class Cursor: def __init__(self): self.buffer = collections.deque() def _prefetch(self): ... async def __aiter__(self): return self async def __anext__(self): if not self.buffer: self.buffer = await self._prefetch() if not self.buffer: raise StopAsyncIteration return self.buffer.popleft() then the ``Cursor`` class can be used as follows:: async for row in Cursor(): print(row) which would be equivalent to the following code:: i = await Cursor().__aiter__() while True: try: row = await i.__anext__() except StopAsyncIteration: break else: print(row) Example 2 ''''''''' The following is a utility class that transforms a regular iterable to an asynchronous one. While this is not a very useful thing to do, the code illustrates the relationship between regular and asynchronous iterators. :: class AsyncIteratorWrapper: def __init__(self, obj): self._it = iter(obj) async def __aiter__(self): return self async def __anext__(self): try: value = next(self._it) except StopIteration: raise StopAsyncIteration return value data = "abc" it = AsyncIteratorWrapper("abc") async for item in it: print(it) Why StopAsyncIteration? ''''''''''''''''''''''' Coroutines are still based on generators internally. So, before PEP 479, there was no fundamental difference between :: def g1(): yield from fut return 'spam' and :: def g2(): yield from fut raise StopIteration('spam') And since PEP 479 is accepted and enabled by default for coroutines, the following example will have its ``StopIteration`` wrapped into a ``RuntimeError`` :: async def a1(): await fut raise StopIteration('spam') The only way to tell the outside code that the iteration has ended is to raise something other than ``StopIteration``. Therefore, a new built-in exception class ``StopAsyncIteration`` was added. Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions raised in coroutines are wrapped in ``RuntimeError``. Debugging Features ------------------ One of the most frequent mistakes that people make when using generators as coroutines is forgetting to use ``yield from``:: @asyncio.coroutine def useful(): asyncio.sleep(1) # this will do noting without 'yield from' For debugging this kind of mistakes there is a special debug mode in asyncio, in which ``@coroutine`` decorator wraps all functions with a special object with a destructor logging a warning. Whenever a wrapped generator gets garbage collected, a detailed logging message is generated with information about where exactly the decorator function was defined, stack trace of where it was collected, etc. Wrapper object also provides a convenient ``__repr__`` function with detailed information about the generator. The only problem is how to enable these debug capabilities. Since debug facilities should be a no-op in production mode, ``@coroutine`` decorator makes the decision of whether to wrap or not to wrap based on an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is possible to run asyncio programs with asyncio's own functions instrumented. ``EventLoop.set_debug``, a different debug facility, has no impact on ``@coroutine`` decorator's behavior. With this proposal, coroutines is a native, distinct from generators, concept. A new method ``set_async_wrapper`` is added to the ``sys`` module, with which frameworks can provide advanced debugging facilities. It is also important to make coroutines as fast and efficient as possible, therefore there are no debug features enabled by default. Example:: async def debug_me(): await asyncio.sleep(1) def async_debug_wrap(generator): return asyncio.AsyncDebugWrapper(generator) sys.set_async_wrapper(async_debug_wrap) debug_me() # <- this line will likely GC the coroutine object and # trigger AsyncDebugWrapper's code. assert isinstance(debug_me(), AsyncDebugWrapper) sys.set_async_wrapper(None) # <- this unsets any previously set wrapper assert not isinstance(debug_me(), AsyncDebugWrapper) If ``sys.set_async_wrapper()`` is called twice, the new wrapper replaces the previous wrapper. ``sys.set_async_wrapper(None)`` unsets the wrapper. Glossary ======== :Coroutine: A coroutine function, or just "coroutine", is declared with ``async def``. It uses ``await`` and ``return value``; see `New Coroutine Declaration Syntax`_ for details. :Coroutine object: Returned from a coroutine function. See `Await Expression`_ for details. :Future-like object: An object with an ``__await__`` method. It is consumed by ``await`` in a coroutine. A coroutine waiting for a Future-like object is suspended until the Future-like object's ``__await__`` completes. ``await`` returns the result of the Future-like object. See `Await Expression`_ for details. :Awaitable: A *future-like* object or a *coroutine object*. See `Await Expression`_ for details. :Generator-based coroutine: Coroutines based in generator syntax. Most common example is ``@asyncio.coroutine``. :Asynchronous context manager: An asynchronous context manager has ``__aenter__`` and ``__aexit__`` methods and can be used with ``async with``. See `Asynchronous Context Managers and "async with"`_ for details. :Asynchronous iterable: An object with an ``__aiter__`` method, which must return an *asynchronous iterator* object. Can be used with ``async for``. See `Asynchronous Iterators and "async for"`_ for details. :Asynchronous iterator: An asynchronous iterator has an ``__anext__`` method.See `Asynchronous Iterators and "async for"`_ for details. List of functions and methods ============================= ================= ======================================= ================= Method Can contain Can't contain ================= ======================================= ================= async def func await, return value yield, yield from async def __a*__ await, return value yield, yield from def __a*__ return Future-like await def __await__ yield, yield from, return iterable await generator yield, yield from, return value await ================= ======================================= ================= Where: * ""async def func": coroutine; * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` defined with the ``async`` keyword; * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` defined without the ``async`` keyword, must return an *awaitable*; * "def __await__": ``__await__`` method to implement *Future-like* objects; * generator: a "regular" generator, function defined with ``def`` and which contains a least one ``yield`` or ``yield from`` expression. *Future-like* is an object with an ``__await__`` method, see `Await Expression`_ section for details. Transition Plan =============== To avoid backwards compatibility issues with ``async`` and ``await`` keywords, it was decided to modify ``tokenizer.c`` in such a way, that it: * recognizes ``async def`` name tokens combination (start of a coroutine); * keeps track of regular functions and coroutines; * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with ``AWAIT`` when in the process of yielding tokens for coroutines. This approach allows for seamless combination of new syntax features (all of them available only in ``async`` functions) with any existing code. An example of having "async def" and "async" attribute in one piece of code:: class Spam: async = 42 async def ham(): print(getattr(Spam, 'async')) # The coroutine can be executed and will print '42' Backwards Compatibility ----------------------- The only backwards incompatible change is an extra argument ``is_async`` to ``FunctionDef`` AST node. But since it is a documented fact that the structure of AST nodes is an implementation detail and subject to change, this should not be considered a serious issue. Grammar Updates --------------- Grammar changes are also fairly minimal:: await_expr: AWAIT test await_stmt: await_expr decorated: decorators (classdef | funcdef | async_funcdef) async_funcdef: ASYNC funcdef async_stmt: ASYNC (funcdef | with_stmt) # will add for_stmt later compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_stmt) atom: ('(' [yield_expr|await_expr|testlist_comp] ')' | '[' [testlist_comp] ']' | '{' [dictorsetmaker] '}' | NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False?) expr_stmt: testlist_star_expr (augassign (yield_expr|await_expr|testlist) | ('=' (yield_expr|await_expr|testlist_star_expr))*) Transition Period Shortcomings ------------------------------ There is just one. Until ``async`` and ``await`` are not proper keywords, it is not possible (or at least very hard) to fix ``tokenizer.c`` to recognize them on the **same line** with ``def`` keyword:: # async and await will always be parsed as variables async def outer(): # 1 def nested(a=(await fut)): pass async def foo(): return (await fut) # 2 Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, a ``SyntaxError`` will be raised. To workaround these issues, the above examples can be easily rewritten to a more readable form:: async def outer(): # 1 a_default = await fut def nested(a=a_default): pass async def foo(): # 2 return (await fut) This limitation will go away as soon as ``async`` and ``await`` ate proper keywords. Or if it's decided to use a future import for this PEP. Deprecation Plans ----------------- ``async`` and ``await`` names will be softly deprecated in CPython 3.5 and 3.6. In 3.7 we will transform them to proper keywords. Making ``async`` and ``await`` proper keywords before 3.7 might make it harder for people to port their code to Python 3. asyncio ------- ``asyncio`` module was adapted and tested to work with coroutines and new statements. Backwards compatibility is 100% preserved. The required changes are mainly: 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.async_def()`` function. 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate ``async()`` function. Design Considerations ===================== No implicit wrapping in Futures ------------------------------- There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key difference is that JavaScript "async functions" always return a Promise. While this approach has some advantages, it also implies that a new Promise object is created on each "async function" invocation. We could implement a similar functionality in Python, by wrapping all coroutines in a Future object, but this has the following disadvantages: 1. Performance. A new Future object would be instantiated on each coroutine call. Moreover, this makes implementation of ``await`` expressions slower (disabling optimizations of ``yield from``). 2. A new built-in ``Future`` object would need to be added. 3. Coming up with a generic ``Future`` interface that is usable for any use case in any framework is a very hard to solve problem. 4. It is not a feature that is used frequently, when most of the code is coroutines. Why "async" and "await" keywords -------------------------------- async/await is not a new concept in programming languages: * C# has it since long time ago [5]_; * proposal to add async/await in ECMAScript 7 [2]_; see also Traceur project [9]_; * Facebook's Hack/HHVM [6]_; * Google's Dart language [7]_; * Scala [8]_; * proposal to add async/await to C++ [10]_; * and many other less popular languages. This is a huge benefit, as some users already have experience with async/await, and because it makes working with many languages in one project easier (Python with ECMAScript 7 for instance). Why "__aiter__" is a coroutine ------------------------------ In principle, ``__aiter__`` could be a regular function. There are several good reasons to make it a coroutine: * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` methods are coroutines, users would often make a mistake defining it as ``async`` anyways; * there might be a need to run some asynchronous operations in ``__aiter__``, for instance to prepare DB queries or do some file operation. Importance of "async" keyword ----------------------------- While it is possible to just implement ``await`` expression and treat all functions with at least one ``await`` as coroutines, this approach makes APIs design, code refactoring and its long time support harder. Let's pretend that Python only has ``await`` keyword:: def useful(): ... await log(...) ... def important(): await useful() If ``useful()`` function is refactored and someone removes all ``await`` expressions from it, it would become a regular python function, and all code that depends on it, including ``important()`` would be broken. To mitigate this issue a decorator similar to ``@asyncio.coroutine`` has to be introduced. Why "async def" --------------- For some people bare ``async name(): pass`` syntax might look more appealing than ``async def name(): pass``. It is certainly easier to type. But on the other hand, it breaks the symmetry between ``async def``, ``async with`` and ``async for``, where ``async`` is a modifier, stating that the statement is asynchronous. It is also more consistent with the existing grammar. Why not a __future__ import --------------------------- ``__future__`` imports are inconvenient and easy to forget to add. Also, they are enabled for the whole source file. Consider that there is a big project with a popular module named "async.py". With future imports it is required to either import it using ``__import__()`` or ``importlib.import_module()`` calls, or to rename the module. The proposed approach makes it possible to continue using old code and modules without a hassle, while coming up with a migration plan for future python versions. Why magic methods start with "a" -------------------------------- New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, and ``__aexit__`` all start with the same prefix "a". An alternative proposal is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. However, to align new magic methods with the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. Why not reuse existing magic names ---------------------------------- An alternative idea about new asynchronous iterators and context managers was to reuse existing magic methods, by adding an ``async`` keyword to their declarations:: class CM: async def __enter__(self): # instead of __aenter__ ... This approach has the following downsides: * it would not be possible to create an object that works in both ``with`` and ``async with`` statements; * it would look confusing and would require some implicit magic behind the scenes in the interpreter; * one of the main points of this proposal is to make coroutines as simple and foolproof as possible. Comprehensions -------------- For the sake of restricting the broadness of this PEP there is no new syntax for asynchronous comprehensions. This should be considered in a separate PEP, if there is a strong demand for this feature. Performance =========== Overall Impact -------------- This proposal introduces no observable performance impact. Here is an output of python's official set of benchmarks [4]_: :: python perf.py -r -b default ../cpython/python.exe ../cpython-aw/python.exe [skipped] Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64 i386 Total CPU cores: 8 ### etree_iterparse ### Min: 0.365359 -> 0.349168: 1.05x faster Avg: 0.396924 -> 0.379735: 1.05x faster Significant (t=9.71) Stddev: 0.01225 -> 0.01277: 1.0423x larger The following not significant results are hidden, use -v to show them: django_v2, 2to3, etree_generate, etree_parse, etree_process, fastpickle, fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. Tokenizer modifications ----------------------- There is no observable slowdown of parsing python files with the modified tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount of time. async/await ----------- The following micro-benchmark was used to determine performance difference between "async" functions and generators:: import sys import time def binary(n): if n <= 0: return 1 l = yield from binary(n - 1) r = yield from binary(n - 1) return l + 1 + r async def abinary(n): if n <= 0: return 1 l = await abinary(n - 1) r = await abinary(n - 1) return l + 1 + r def timeit(gen, depth, repeat): t0 = time.time() for _ in range(repeat): list(gen(depth)) t1 = time.time() print('{}({}) * {}: total {:.3f}s'.format( gen.__name__, depth, repeat, t1-t0)) The result is that there is no observable performance difference. Minimum timing of 3 runs :: abinary(19) * 30: total 12.985s binary(19) * 30: total 12.953s Note that depth of 19 means 1,048,575 calls. Reference Implementation ======================== The reference implementation can be found here: [3]_. List of high-level changes and new protocols -------------------------------------------- 1. New syntax for defining coroutines: ``async def`` and new ``await`` keyword. 2. New ``__await__`` method for Future-like objects. 3. New syntax for asynchronous context managers: ``async with``. And associated protocol with ``__aenter__`` and ``__aexit__`` methods. 4. New syntax for asynchronous iteration: ``async for``. And associated protocol with ``__aiter__``, ``__aexit__`` and new built-in exception ``StopAsyncIteration``. 5. New AST nodes: ``AsyncFor``, ``AsyncWith``, ``Await``; ``FunctionDef`` AST node got a new argument ``is_async``. 6. New functions: ``sys.set_async_wrapper(callback)`` and ``types.async_def(gen)``. 7. New ``CO_ASYNC`` bit flag for code objects. While the list of changes and new things is not short, it is important to understand, that most users will not use these features directly. It is intended to be used in frameworks and libraries to provide users with convenient to use and unambiguous APIs with ``async def``, ``await``, ``async for`` and ``async with`` syntax. Working example --------------- All concepts proposed in this PEP are implemented [3]_ and can be tested. :: import asyncio async def echo_server(): print('Serving on localhost:8000') await asyncio.start_server(handle_connection, 'localhost', 8000) async def handle_connection(reader, writer): print('New connection...') while True: data = await reader.read(8192) if not data: break print('Sending {:.10}... back'.format(repr(data))) writer.write(data) loop = asyncio.get_event_loop() loop.run_until_complete(echo_server()) try: loop.run_forever() finally: loop.close() References ========== .. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions .. [3] https://github.com/1st1/cpython/tree/await .. [4] https://hg.python.org/benchmarks .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx .. [6] http://docs.hhvm.com/manual/en/hack.async.php .. [7] https://www.dartlang.org/articles/await-async/ .. [8] http://docs.scala-lang.org/sips/pending/async.html .. [9] https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) Acknowledgments =============== I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew Svetlov, and ?ukasz Langa for their initial feedback. Copyright ========= This document has been placed in the public domain. From guido at python.org Fri Apr 17 22:07:27 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 13:07:27 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) discussion moved to python-dev Message-ID: I've posted a new draft of PEP 484, but in order to reach the right audience from now on I'm posting to python-dev. See you there! Please don't reply here, it's best to have all discussion in one place. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Apr 17 22:19:42 2015 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 17 Apr 2015 20:19:42 +0000 (UTC) Subject: [Python-ideas] async/await in Python References: <553157F2.8060408@gmail.com> Message-ID: Hi, Yury Selivanov writes: > Here's my proposal to add async/await in Python. As a contributor to asyncio, I'm a strong supporter of this PEP. IMO it should land into Python 3.5 to confirm that Python promotes asynchronous programming (and is the best language for async programming? :-)) and accelerate the adoption of asyncio "everywhere". The first time Yury shared with me his idea of new keywords (at Pycon Montreal 2014), I understood that it was just syntax sugar. In fact, it adds also "async for" and "async with" which adds new features. They are almost required to make asyncio usage easier. "async for" would help database ORMs or the aiofiles project (run file I/O in threads). "async with" helps also ORMs. I already reviewed this PEP before Yury sent it to this list, so sorry, I have no longer major issue to report :-) > Debugging Features > ------------------ > (...) > With this proposal, coroutines is a native, distinct from generators, > concept. A new method ``set_async_wrapper`` is added to the ``sys`` module, I suggest to rename it to set_coroutine_wrapper() to be more consistent with the PEP. By the way, asyncio has a private "CoroWrapper" class (coroutine wrapper). It is the class used in debug mode to emit warnings described in this section. I also suggest to add get_coroutine_wrapper(), similar to sys.get/set_trace(). It is required for a framework to check if the wrapper is correctly setup (and no concurrent framework installed its own wrapper?). > List of functions and methods > ============================= > > ================= ======================================= ================= > Method Can contain Can't contain > ================= ======================================= ================= > async def func await, return value yield, yield from > async def __a*__ await, return value yield, yield from > def __a*__ return Future-like await > def __await__ yield, yield from, return iterable await > generator yield, yield from, return value await > ================= ======================================= ================= You may add asyncio coroutines, which are special (using yield is invalid), for the comparison: can contain: yield from, return value can't contain: yield, await > Backwards Compatibility > ----------------------- > > The only backwards incompatible change is an extra argument ``is_async`` to > ``FunctionDef`` AST node. But since it is a documented fact that the > structure > of AST nodes is an implementation detail and subject to change, this > should not > be considered a serious issue. The PEP breaks the backward compatibility in Python 3.7 when async and await will become keywords. You may mention it. > Why not a __future__ import > --------------------------- > > ``__future__`` imports are inconvenient and easy to forget to add. (...) I understand that you describe a situation where "async def ..." would raise a SyntaxError when the __future__ is missing. I think that an *optional* __future__ would help projects to be prepared for async & await keywords. I mean, you can add "from __future__ import async" to get async & await as keywords in the current file, as we did before fore "from __future__ import with_statement". Also, I'm not opposed to require "from __future__ import async" to get the new feature. I don't expect async/await to popup in *all* files of a project, so changes are limited. I would avoid border case issues in the parser (issues described above: when async/await is used in parameters or one a single line). Nowadays, there are more and more linters. For example, in the OpenStack project, an issue detected by static analyzers ("hacking" based on flake8) must be fixed by the author of a patch, otherwise the patch cannot be merged. It's just mandatory. So maybe __future__ is less important than it was when with was added in Python 2.6? > Why magic methods start with "a" > -------------------------------- > > New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, > and ``__aexit__`` all start with the same prefix "a". An alternative > proposal > is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. I prefer __async_iter__ over __aiter__ (and __async_next/enter/exit__), but I can survive to write __aiter__. __async_iter__ is more consistent with "async" prefix added to functions (async def). > However, to align new magic methods with the existing ones, such as > ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. I don't think that past mistakes justify new mistakes :-D Victor From contact at ionelmc.ro Fri Apr 17 22:45:54 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Fri, 17 Apr 2015 23:45:54 +0300 Subject: [Python-ideas] Fix that broken callable builtin Message-ID: Hello, I had an issue today with the `callable` builtin because it doesn't correctly check that the object has the __call__ attribute. Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but that's not very straightforward. A more straightforward implementation would do something like `hasattr(a, '__call__')`. For example: Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 > bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> callable > > >>> class A: > ... @property > ... def __call__(self): > ... raise AttributeError('go away') > ... > >>> a = A() > >>> a > <__main__.A object at 0x000000000365B5C0> > >>> a.__call__ > Traceback (most recent call last): > File "", line 1, in > File "", line 4, in __call__ > AttributeError: go away > >>> callable(a) > True > >>> # it should be False :( > ?So it boils down to this: > >>> hasattr(a, "__call__") > False > >>> hasattr(type(a), "__call__") > True My issue is that I didn't call `callable(type(a))` but just `callable(a)`. Clearly mismatching what happens when you do hasattr(a, "__call__"). To put in contrast, this is legal and clearly indicates the descriptors are being used as expected: >>> class B: > ... @property > ... def __call__(self): > ... return lambda: 1 > ... > >>> b = B() > >>> b() > 1 > ?There? ?'s some more discussing in issue 23990 ? ? where I get slightly angry, sorry.? ?So were is this change actually useful? Proxies! Since new-style objects in Python you cannot really proxy the callable aspect of objects, because `callable` just checks that a field is set in a C struct.? ? This is fairly inconvenient because you have to know upfront if your target is going to be callable or not.? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Apr 17 22:55:17 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 13:55:17 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: I think you're fighting windmills. Like most special operations (e.g. __add__), a __call__ attribute on the object does not work, i.e. it does not make the object callable. E.g. $ python3 Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> class C: pass ... >>> c = C() >>> c.__call__ = lambda *a: a >>> c() Traceback (most recent call last): File "", line 1, in TypeError: 'C' object is not callable >>> callable(c) False >>> hasattr(c, '__call__') True >>> On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian M?rie? wrote: > Hello, > > I had an issue today with the `callable` builtin because it doesn't > correctly check that the object has the __call__ attribute. > > Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but > that's not very straightforward. A more straightforward implementation > would do something like `hasattr(a, '__call__')`. > > For example: > > Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 >> bit (AMD64)] on win32 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> callable >> >> >>> class A: >> ... @property >> ... def __call__(self): >> ... raise AttributeError('go away') >> ... >> >>> a = A() >> >>> a >> <__main__.A object at 0x000000000365B5C0> >> >>> a.__call__ >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 4, in __call__ >> AttributeError: go away >> >>> callable(a) >> True >> >>> # it should be False :( >> > > ?So it boils down to this: > >> >>> hasattr(a, "__call__") >> False >> >>> hasattr(type(a), "__call__") >> True > > > My issue is that I didn't call `callable(type(a))` but just `callable(a)`. > Clearly mismatching what happens when you do hasattr(a, "__call__"). > > To put in contrast, this is legal and clearly indicates the descriptors > are being used as expected: > > >>> class B: >> ... @property >> ... def __call__(self): >> ... return lambda: 1 >> ... >> >>> b = B() >> >>> b() >> 1 >> > > ?There? > ?'s some more discussing in issue 23990 > ? > ? where I get slightly angry, sorry.? > > > ?So were is this change actually useful? Proxies! Since new-style objects > in Python you cannot really proxy the callable aspect of objects, because > `callable` just checks that a field is set in a C struct.? > ? This is fairly inconvenient because you have to know upfront if your > target is going to be callable or not.? > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Fri Apr 17 23:04:33 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 00:04:33 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: Well yes, from that example it look right, because the call operator uses the __call__ attribute from the type of the object. However, when the call operator gets the __call__ method it will actually use it as a descriptor. >From that perspective it's inconsistent. Also there's the issue about not being able to implement a true proxy (as outlined before). What actually prevents this being fixed? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum wrote: > I think you're fighting windmills. Like most special operations (e.g. > __add__), a __call__ attribute on the object does not work, i.e. it does > not make the object callable. E.g. > > $ python3 > Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) > [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> class C: pass > ... > >>> c = C() > >>> c.__call__ = lambda *a: a > >>> c() > Traceback (most recent call last): > File "", line 1, in > TypeError: 'C' object is not callable > >>> callable(c) > False > >>> hasattr(c, '__call__') > True > >>> > > On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian M?rie? > wrote: > >> Hello, >> >> I had an issue today with the `callable` builtin because it doesn't >> correctly check that the object has the __call__ attribute. >> >> Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` but >> that's not very straightforward. A more straightforward implementation >> would do something like `hasattr(a, '__call__')`. >> >> For example: >> >> Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 >>> bit (AMD64)] on win32 >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> callable >>> >>> >>> class A: >>> ... @property >>> ... def __call__(self): >>> ... raise AttributeError('go away') >>> ... >>> >>> a = A() >>> >>> a >>> <__main__.A object at 0x000000000365B5C0> >>> >>> a.__call__ >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "", line 4, in __call__ >>> AttributeError: go away >>> >>> callable(a) >>> True >>> >>> # it should be False :( >>> >> >> ?So it boils down to this: >> >>> >>> hasattr(a, "__call__") >>> False >>> >>> hasattr(type(a), "__call__") >>> True >> >> >> My issue is that I didn't call `callable(type(a))` but just >> `callable(a)`. Clearly mismatching what happens when you do hasattr(a, >> "__call__"). >> >> To put in contrast, this is legal and clearly indicates the descriptors >> are being used as expected: >> >> >>> class B: >>> ... @property >>> ... def __call__(self): >>> ... return lambda: 1 >>> ... >>> >>> b = B() >>> >>> b() >>> 1 >>> >> >> ?There? >> ?'s some more discussing in issue 23990 >> ? >> ? where I get slightly angry, sorry.? >> >> >> ?So were is this change actually useful? Proxies! Since new-style objects >> in Python you cannot really proxy the callable aspect of objects, because >> `callable` just checks that a field is set in a C struct.? >> ? This is fairly inconvenient because you have to know upfront if your >> target is going to be callable or not.? >> >> >> >> Thanks, >> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Fri Apr 17 23:09:28 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 17:09:28 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: <55317688.8010904@gmail.com> Hi Victor, Thanks for another review ;) On 2015-04-17 4:19 PM, Victor Stinner wrote: > Hi, > > Yury Selivanov writes: >> Here's my proposal to add async/await in Python. > As a contributor to asyncio, I'm a strong supporter of this PEP. IMO it > should land into Python 3.5 to confirm that Python promotes asynchronous > programming (and is the best language for async programming? :-)) and > accelerate the adoption of asyncio "everywhere". > > The first time Yury shared with me his idea of new keywords (at Pycon > Montreal 2014), I understood that it was just syntax sugar. In fact, it adds > also "async for" and "async with" which adds new features. They are almost > required to make asyncio usage easier. "async for" would help database ORMs > or the aiofiles project (run file I/O in threads). "async with" helps also ORMs. > > I already reviewed this PEP before Yury sent it to this list, so sorry, I > have no longer major issue to report :-) > >> Debugging Features >> ------------------ >> (...) >> With this proposal, coroutines is a native, distinct from generators, >> concept. A new method ``set_async_wrapper`` is added to the ``sys`` module, > I suggest to rename it to set_coroutine_wrapper() to be more consistent with > the PEP. By the way, asyncio has a private "CoroWrapper" class (coroutine > wrapper). It is the class used in debug mode to emit warnings described in > this section. I like "set_coroutine_wrapper()" name, it's better. I'll update the PEP. > > I also suggest to add get_coroutine_wrapper(), similar to > sys.get/set_trace(). It is required for a framework to check if the wrapper > is correctly setup (and no concurrent framework installed its own wrapper?). > >> List of functions and methods >> ============================= >> >> ================= ======================================= ================= >> Method Can contain Can't contain >> ================= ======================================= ================= >> async def func await, return value yield, yield from >> async def __a*__ await, return value yield, yield from >> def __a*__ return Future-like await >> def __await__ yield, yield from, return iterable await >> generator yield, yield from, return value await >> ================= ======================================= ================= > You may add asyncio coroutines, which are special (using yield is invalid), > for the comparison: > > can contain: yield from, return value > can't contain: yield, await > >> Backwards Compatibility >> ----------------------- >> >> The only backwards incompatible change is an extra argument ``is_async`` to >> ``FunctionDef`` AST node. But since it is a documented fact that the >> structure >> of AST nodes is an implementation detail and subject to change, this >> should not >> be considered a serious issue. > The PEP breaks the backward compatibility in Python 3.7 when async and await > will become keywords. You may mention it. > > >> Why not a __future__ import >> --------------------------- >> >> ``__future__`` imports are inconvenient and easy to forget to add. (...) > I understand that you describe a situation where "async def ..." would raise > a SyntaxError when the __future__ is missing. > > I think that an *optional* __future__ would help projects to be prepared for > async & await keywords. > > I mean, you can add "from __future__ import async" to get async & await as > keywords in the current file, as we did before fore "from __future__ import > with_statement". > > Also, I'm not opposed to require "from __future__ import async" to get the > new feature. I don't expect async/await to popup in *all* files of a > project, so changes are limited. I would avoid border case issues in the > parser (issues described above: when async/await is used in parameters or > one a single line). > > Nowadays, there are more and more linters. For example, in the OpenStack > project, an issue detected by static analyzers ("hacking" based on flake8) > must be fixed by the author of a patch, otherwise the patch cannot be > merged. It's just mandatory. So maybe __future__ is less important than it > was when with was added in Python 2.6? > >> Why magic methods start with "a" >> -------------------------------- >> >> New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, >> and ``__aexit__`` all start with the same prefix "a". An alternative >> proposal >> is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. > I prefer __async_iter__ over __aiter__ (and __async_next/enter/exit__), but > I can survive to write __aiter__. __async_iter__ is more consistent with > "async" prefix added to functions (async def). > > >> However, to align new magic methods with the existing ones, such as >> ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > I don't think that past mistakes justify new mistakes :-D > > Victor > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Fri Apr 17 23:10:30 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 17 Apr 2015 14:10:30 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: <20150417211030.GE28035@stoneleaf.us> On 04/18, Ionel Cristian M?rie? wrote: > Also there's the issue about not being able to implement a true proxy (as > outlined before). Proxies are a bit of a pain. But you can create your own callable function. Something like (untested): def callable(obj): try: func = obj.__call__ return True except AttributeError: return False -- ~Ethan~ From guido at python.org Fri Apr 17 23:15:57 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 14:15:57 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: You won't have any more luck defining __add__ as a property -- just don't do that. On how to implement a proxy, I'll let other explain. But this is not it. On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian M?rie? wrote: > Well yes, from that example it look right, because the call operator uses > the __call__ attribute from the type of the object. However, when the call > operator gets the __call__ method it will actually use it as a descriptor. > From that perspective it's inconsistent. > > Also there's the issue about not being able to implement a true proxy (as > outlined before). > > What actually prevents this being fixed? > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum > wrote: > >> I think you're fighting windmills. Like most special operations (e.g. >> __add__), a __call__ attribute on the object does not work, i.e. it does >> not make the object callable. E.g. >> >> $ python3 >> Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) >> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >> >>> class C: pass >> ... >> >>> c = C() >> >>> c.__call__ = lambda *a: a >> >>> c() >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: 'C' object is not callable >> >>> callable(c) >> False >> >>> hasattr(c, '__call__') >> True >> >>> >> >> On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian M?rie? < >> contact at ionelmc.ro> wrote: >> >>> Hello, >>> >>> I had an issue today with the `callable` builtin because it doesn't >>> correctly check that the object has the __call__ attribute. >>> >>> Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` >>> but that's not very straightforward. A more straightforward implementation >>> would do something like `hasattr(a, '__call__')`. >>> >>> For example: >>> >>> Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 >>>> bit (AMD64)] on win32 >>>> Type "help", "copyright", "credits" or "license" for more information. >>>> >>> callable >>>> >>>> >>> class A: >>>> ... @property >>>> ... def __call__(self): >>>> ... raise AttributeError('go away') >>>> ... >>>> >>> a = A() >>>> >>> a >>>> <__main__.A object at 0x000000000365B5C0> >>>> >>> a.__call__ >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> File "", line 4, in __call__ >>>> AttributeError: go away >>>> >>> callable(a) >>>> True >>>> >>> # it should be False :( >>>> >>> >>> ?So it boils down to this: >>> >>>> >>> hasattr(a, "__call__") >>>> False >>>> >>> hasattr(type(a), "__call__") >>>> True >>> >>> >>> My issue is that I didn't call `callable(type(a))` but just >>> `callable(a)`. Clearly mismatching what happens when you do hasattr(a, >>> "__call__"). >>> >>> To put in contrast, this is legal and clearly indicates the descriptors >>> are being used as expected: >>> >>> >>> class B: >>>> ... @property >>>> ... def __call__(self): >>>> ... return lambda: 1 >>>> ... >>>> >>> b = B() >>>> >>> b() >>>> 1 >>>> >>> >>> ?There? >>> ?'s some more discussing in issue 23990 >>> ? >>> ? where I get slightly angry, sorry.? >>> >>> >>> ?So were is this change actually useful? Proxies! Since new-style >>> objects in Python you cannot really proxy the callable aspect of objects, >>> because `callable` just checks that a field is set in a C struct.? >>> ? This is fairly inconvenient because you have to know upfront if your >>> target is going to be callable or not.? >>> >>> >>> >>> Thanks, >>> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Fri Apr 17 23:19:39 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 00:19:39 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: __add__ as a property/descriptor seems to work fine, eg: >>> class C: > ... @property > ... def __add__(self): > ... return lambda other: [self, other] > ... > >>> C() + C() > [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at > 0x0000000003652CC0>] > Am I missing something? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:15 AM, Guido van Rossum wrote: > You won't have any more luck defining __add__ as a property -- just don't > do that. > > On how to implement a proxy, I'll let other explain. But this is not it. > > On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian M?rie? > wrote: > >> Well yes, from that example it look right, because the call operator uses >> the __call__ attribute from the type of the object. However, when the call >> operator gets the __call__ method it will actually use it as a descriptor. >> From that perspective it's inconsistent. >> >> Also there's the issue about not being able to implement a true proxy (as >> outlined before). >> >> What actually prevents this being fixed? >> >> >> Thanks, >> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >> >> On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum >> wrote: >> >>> I think you're fighting windmills. Like most special operations (e.g. >>> __add__), a __call__ attribute on the object does not work, i.e. it does >>> not make the object callable. E.g. >>> >>> $ python3 >>> Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) >>> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> class C: pass >>> ... >>> >>> c = C() >>> >>> c.__call__ = lambda *a: a >>> >>> c() >>> Traceback (most recent call last): >>> File "", line 1, in >>> TypeError: 'C' object is not callable >>> >>> callable(c) >>> False >>> >>> hasattr(c, '__call__') >>> True >>> >>> >>> >>> On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian M?rie? < >>> contact at ionelmc.ro> wrote: >>> >>>> Hello, >>>> >>>> I had an issue today with the `callable` builtin because it doesn't >>>> correctly check that the object has the __call__ attribute. >>>> >>>> Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` >>>> but that's not very straightforward. A more straightforward implementation >>>> would do something like `hasattr(a, '__call__')`. >>>> >>>> For example: >>>> >>>> Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 >>>>> 64 bit (AMD64)] on win32 >>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>> >>> callable >>>>> >>>>> >>> class A: >>>>> ... @property >>>>> ... def __call__(self): >>>>> ... raise AttributeError('go away') >>>>> ... >>>>> >>> a = A() >>>>> >>> a >>>>> <__main__.A object at 0x000000000365B5C0> >>>>> >>> a.__call__ >>>>> Traceback (most recent call last): >>>>> File "", line 1, in >>>>> File "", line 4, in __call__ >>>>> AttributeError: go away >>>>> >>> callable(a) >>>>> True >>>>> >>> # it should be False :( >>>>> >>>> >>>> ?So it boils down to this: >>>> >>>>> >>> hasattr(a, "__call__") >>>>> False >>>>> >>> hasattr(type(a), "__call__") >>>>> True >>>> >>>> >>>> My issue is that I didn't call `callable(type(a))` but just >>>> `callable(a)`. Clearly mismatching what happens when you do hasattr(a, >>>> "__call__"). >>>> >>>> To put in contrast, this is legal and clearly indicates the descriptors >>>> are being used as expected: >>>> >>>> >>> class B: >>>>> ... @property >>>>> ... def __call__(self): >>>>> ... return lambda: 1 >>>>> ... >>>>> >>> b = B() >>>>> >>> b() >>>>> 1 >>>>> >>>> >>>> ?There? >>>> ?'s some more discussing in issue 23990 >>>> ? >>>> ? where I get slightly angry, sorry.? >>>> >>>> >>>> ?So were is this change actually useful? Proxies! Since new-style >>>> objects in Python you cannot really proxy the callable aspect of objects, >>>> because `callable` just checks that a field is set in a C struct.? >>>> ? This is fairly inconvenient because you have to know upfront if your >>>> target is going to be callable or not.? >>>> >>>> >>>> >>>> Thanks, >>>> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Apr 17 23:34:07 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 Apr 2015 23:34:07 +0200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: <55317C4F.2050207@egenix.com> On 17.04.2015 20:58, Yury Selivanov wrote: > Hello python-ideas, > > Here's my proposal to add async/await in Python. > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > and I hope that this PEP can be another big step. Who knows, maybe it > will be one of the reasons to drive people's interest towards Python 3. Thanks for proposing this syntax. With async/await added to Python I think I would actually start writing native async code instead of relying on gevent to do all the heavy lifting for me ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 17 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guido at python.org Fri Apr 17 23:39:10 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 14:39:10 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: I think you've found an unintended and undocumented backdoor. I admit I don't understand how this works in CPython. Overloaded operators like __add__ or __call__ should be methods in the class, and we don't look for them in the instance. But somehow defining them with @property works (I guess because @property is in the class). What's different for __call__ is that callable() exists. And this is probably why I exorcised it Python 3.0 -- but apparently it's back. :-( In the end callable() doesn't always produce a correct answer; but maybe we can make it work in this case by first testing the class and then the instance? Something like (untested): def callable(x): return hasattr(x.__class__, '__call__') and hasattr(x, '__call__') On Fri, Apr 17, 2015 at 2:19 PM, Ionel Cristian M?rie? wrote: > __add__ as a property/descriptor seems to work fine, eg: > > >>> class C: >> ... @property >> ... def __add__(self): >> ... return lambda other: [self, other] >> ... >> >>> C() + C() >> [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at >> 0x0000000003652CC0>] >> > > Am I missing something? > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > On Sat, Apr 18, 2015 at 12:15 AM, Guido van Rossum > wrote: > >> You won't have any more luck defining __add__ as a property -- just don't >> do that. >> >> On how to implement a proxy, I'll let other explain. But this is not it. >> >> On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian M?rie? < >> contact at ionelmc.ro> wrote: >> >>> Well yes, from that example it look right, because the call operator >>> uses the __call__ attribute from the type of the object. However, when the >>> call operator gets the __call__ method it will actually use it as a >>> descriptor. From that perspective it's inconsistent. >>> >>> Also there's the issue about not being able to implement a true proxy >>> (as outlined before). >>> >>> What actually prevents this being fixed? >>> >>> >>> Thanks, >>> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >>> >>> On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum >>> wrote: >>> >>>> I think you're fighting windmills. Like most special operations (e.g. >>>> __add__), a __call__ attribute on the object does not work, i.e. it does >>>> not make the object callable. E.g. >>>> >>>> $ python3 >>>> Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) >>>> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin >>>> Type "help", "copyright", "credits" or "license" for more information. >>>> >>> class C: pass >>>> ... >>>> >>> c = C() >>>> >>> c.__call__ = lambda *a: a >>>> >>> c() >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> TypeError: 'C' object is not callable >>>> >>> callable(c) >>>> False >>>> >>> hasattr(c, '__call__') >>>> True >>>> >>> >>>> >>>> On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian M?rie? < >>>> contact at ionelmc.ro> wrote: >>>> >>>>> Hello, >>>>> >>>>> I had an issue today with the `callable` builtin because it doesn't >>>>> correctly check that the object has the __call__ attribute. >>>>> >>>>> Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` >>>>> but that's not very straightforward. A more straightforward implementation >>>>> would do something like `hasattr(a, '__call__')`. >>>>> >>>>> For example: >>>>> >>>>> Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 >>>>>> 64 bit (AMD64)] on win32 >>>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>>> >>> callable >>>>>> >>>>>> >>> class A: >>>>>> ... @property >>>>>> ... def __call__(self): >>>>>> ... raise AttributeError('go away') >>>>>> ... >>>>>> >>> a = A() >>>>>> >>> a >>>>>> <__main__.A object at 0x000000000365B5C0> >>>>>> >>> a.__call__ >>>>>> Traceback (most recent call last): >>>>>> File "", line 1, in >>>>>> File "", line 4, in __call__ >>>>>> AttributeError: go away >>>>>> >>> callable(a) >>>>>> True >>>>>> >>> # it should be False :( >>>>>> >>>>> >>>>> ?So it boils down to this: >>>>> >>>>>> >>> hasattr(a, "__call__") >>>>>> False >>>>>> >>> hasattr(type(a), "__call__") >>>>>> True >>>>> >>>>> >>>>> My issue is that I didn't call `callable(type(a))` but just >>>>> `callable(a)`. Clearly mismatching what happens when you do hasattr(a, >>>>> "__call__"). >>>>> >>>>> To put in contrast, this is legal and clearly indicates the >>>>> descriptors are being used as expected: >>>>> >>>>> >>> class B: >>>>>> ... @property >>>>>> ... def __call__(self): >>>>>> ... return lambda: 1 >>>>>> ... >>>>>> >>> b = B() >>>>>> >>> b() >>>>>> 1 >>>>>> >>>>> >>>>> ?There? >>>>> ?'s some more discussing in issue 23990 >>>>> ? >>>>> ? where I get slightly angry, sorry.? >>>>> >>>>> >>>>> ?So were is this change actually useful? Proxies! Since new-style >>>>> objects in Python you cannot really proxy the callable aspect of objects, >>>>> because `callable` just checks that a field is set in a C struct.? >>>>> ? This is fairly inconvenient because you have to know upfront if your >>>>> target is going to be callable or not.? >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>> >>>> >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>> >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Apr 17 23:41:53 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 14:41:53 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <55317C4F.2050207@egenix.com> References: <553157F2.8060408@gmail.com> <55317C4F.2050207@egenix.com> Message-ID: On Fri, Apr 17, 2015 at 2:34 PM, M.-A. Lemburg wrote: > On 17.04.2015 20:58, Yury Selivanov wrote: > > Hello python-ideas, > > > > Here's my proposal to add async/await in Python. > > > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > > and I hope that this PEP can be another big step. Who knows, maybe it > > will be one of the reasons to drive people's interest towards Python 3. > > Thanks for proposing this syntax. With async/await added to Python I think > I > would actually start writing native async code instead of relying on > gevent to do all the heavy lifting for me ;-) > A most ringing endorsement. (I'm in favor of Yuri's PEP too, but I don't want to push too hard for it as I haven't had the time to consider all the consequences. PS. It's now live as PEP 492: https://www.python.org/dev/peps/pep-0492/ -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Apr 17 23:44:47 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 17 Apr 2015 14:44:47 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: <20150417214447.GF28035@stoneleaf.us> On 04/18, Ionel Cristian M?rie? wrote: > __add__ as a property/descriptor seems to work fine, eg: > > >>> class C: > > ... @property > > ... def __add__(self): > > ... return lambda other: [self, other] > > ... > > >>> C() + C() > > [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at > > 0x0000000003652CC0>] > > > > Am I missing something? What happens when your __add__ raises an AttributeError? -- ~Ethan~ From contact at ionelmc.ro Fri Apr 17 23:21:51 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 00:21:51 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: <20150417211030.GE28035@stoneleaf.us> References: <20150417211030.GE28035@stoneleaf.us> Message-ID: Yes indeed, that's one way but I wouldn't want to monkeypatch the `callable` builtin. People wouldn't expect that library would even dare monkeypatch builtin. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman wrote: > On 04/18, Ionel Cristian M?rie? wrote: > > > Also there's the issue about not being able to implement a true proxy (as > > outlined before). > > Proxies are a bit of a pain. But you can create your own callable > function. > > Something like (untested): > > def callable(obj): > try: > func = obj.__call__ > return True > except AttributeError: > return False > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 18 00:00:03 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Apr 2015 08:00:03 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov wrote: > The following new syntax is used to declare a coroutine:: > > async def read_data(db): > pass > > Key properties of coroutines: > > * Coroutines are always generators, even if they do not contain ``await`` > expressions. Do you mean that a coroutine is a special form of generator, or that a function declared with "async" is always a coroutine even if it doesn't contain an "await"? You're separating coroutines from generators in many ways, so maybe I'm just misreading this. > Asynchronous Context Managers and "async with" > ---------------------------------------------- > > An *asynchronous context manager* is a context manager that is able to > suspend > execution in its *enter* and *exit* methods. A regular 'with' block guarantees that subsequent code won't execute until __exit__ completes. I presume the same guarantee is applied to an 'async with' block, and thus it can be used only inside a coroutine? (Edit: Oops, missed where that's said further down - SyntaxError to use it outside a coroutine. Might still be worth clarifying the guarantee somewhere, otherwise ignore me.) > New Syntax > '''''''''' > > A new statement for iterating through asynchronous iterators is proposed:: > > async for TARGET in ITER: > BLOCK > else: > BLOCK2 > > which is semantically equivalent to:: > > iter = (ITER) > iter = await type(iter).__aiter__(iter) > running = True > while running: > try: > TARGET = await type(iter).__anext__(iter) > except StopAsyncIteration: > running = False > else: > BLOCK > else: > BLOCK2 (Not sure why you don't just use "break" instead of "running = False"? Maybe I'm blind to some distinction here.) > Why StopAsyncIteration? > ''''''''''''''''''''''' > > Coroutines are still based on generators internally. So, before PEP 479, > there > was no fundamental difference between > > :: > > def g1(): > yield from fut > return 'spam' > > and > > :: > > def g2(): > yield from fut > raise StopIteration('spam') > > And since PEP 479 is accepted and enabled by default for coroutines, the > following example will have its ``StopIteration`` wrapped into a > ``RuntimeError`` > > :: > > async def a1(): > await fut > raise StopIteration('spam') > > The only way to tell the outside code that the iteration has ended is to > raise > something other than ``StopIteration``. Therefore, a new built-in exception > class ``StopAsyncIteration`` was added. > > Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions > raised > in coroutines are wrapped in ``RuntimeError``. Can't a coroutine simply use 'return'? I'm not hugely familiar with them, but I'm not understanding here why you need a completely separate exception - and, more importantly, why you wouldn't have the same consideration of "StopAsyncIteration leakage". It ought to be possible for the coroutine to return, StopIteration be synthesized, and the __anext__ function to bubble that out - exactly the way it does for generators today. > Glossary > ======== > > :Coroutine: > A coroutine function, or just "coroutine", is declared with ``async > def``. > It uses ``await`` and ``return value``; see `New Coroutine Declaration > Syntax`_ for details. (Using this definition to justify my statement above) > Why not a __future__ import > --------------------------- > > ``__future__`` imports are inconvenient and easy to forget to add. Also, > they > are enabled for the whole source file. Consider that there is a big project > with a popular module named "async.py". With future imports it is required > to > either import it using ``__import__()`` or ``importlib.import_module()`` > calls, > or to rename the module. The proposed approach makes it possible to > continue > using old code and modules without a hassle, while coming up with a > migration > plan for future python versions. The clash would occur only in modules that use the __future__ import, wouldn't it? And those modules are going to have to be fixed by version X.Y anyway - the whole point of the __future__ import is to detect that. > Why magic methods start with "a" > -------------------------------- > > New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, > and ``__aexit__`` all start with the same prefix "a". An alternative > proposal > is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. > However, to align new magic methods with the existing ones, such as > ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. I support the short names, with the possible exception that "__await__" now looks like "asynchronous wait", which may or may not be desirable. > Comprehensions > -------------- > > For the sake of restricting the broadness of this PEP there is no new syntax > for asynchronous comprehensions. This should be considered in a separate > PEP, > if there is a strong demand for this feature. Likewise asynchronous lambda functions (or maybe I just missed it). Overall comment: The changes appear to be infecting a lot of code. Will this suddenly be something that everyone has to think about, or can it still be ignored in projects that don't use it? There'll be a few object types that would benefit from becoming async context managers as well as regular context managers (I would expect, for instance, that the built-in file type would allow its auto-closing to be asynchronous), but if a third-party module completely ignores this proposal, everything should continue to work, right? I'm seeing the potential for a lot of code duplication here, but possibly once things settle into real-world usage, a few easy decorators will help out with that (like the total_ordering decorator for handling __gt__, __le__, etc etc). +1 on the proposal as a whole. ChrisA From guido at python.org Sat Apr 18 00:00:03 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 15:00:03 -0700 Subject: [Python-ideas] PEP 484 (Type Hints) discussion moved to python-dev In-Reply-To: References: Message-ID: (Sorry, that was a little premature. I've now sent the message to python-dev and pushed the new PEP text to the peps repo. python.org will follow soon.) On Fri, Apr 17, 2015 at 1:07 PM, Guido van Rossum wrote: > I've posted a new draft of PEP 484, but in order to reach the right > audience from now on I'm posting to python-dev. See you there! Please don't > reply here, it's best to have all discussion in one place. > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Sat Apr 18 00:14:18 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 01:14:18 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: <20150417214447.GF28035@stoneleaf.us> References: <20150417214447.GF28035@stoneleaf.us> Message-ID: Same as what would happen when you use the call operator on an object that has an AttributeError raising property: >>> class D: > ... @property > ... def __add__(self): > ... raise AttributeError('Not so fast, pardner!') > ... > >>> D() + D() > Traceback (most recent call last): > File "", line 1, in > File "", line 4, in __add__ > AttributeError: Not so fast, pardner! Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:44 AM, Ethan Furman wrote: > On 04/18, Ionel Cristian M?rie? wrote: > > > __add__ as a property/descriptor seems to work fine, eg: > > > > >>> class C: > > > ... @property > > > ... def __add__(self): > > > ... return lambda other: [self, other] > > > ... > > > >>> C() + C() > > > [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at > > > 0x0000000003652CC0>] > > > > > > > Am I missing something? > > What happens when your __add__ raises an AttributeError? > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Apr 18 00:15:28 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 17 Apr 2015 15:15:28 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: Could you show the best yield from parallel for the example at the end, so it's easier to see the readability benefits? Or, even better, provide a simple example that uses async for or async with and the yield from parallel, so it's easier to see the (I'm assuming even bigger, by a long shot) readability benefits? Sent from my iPhone > On Apr 17, 2015, at 11:58, Yury Selivanov wrote: > > Hello python-ideas, > > Here's my proposal to add async/await in Python. > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > and I hope that this PEP can be another big step. Who knows, maybe it > will be one of the reasons to drive people's interest towards Python 3. > > > PEP: XXX > Title: Coroutines with async and await syntax > Version: $Revision$ > Last-Modified: $Date$ > Author: Yury Selivanov > Discussions-To: Python-Dev > Python-Version: 3.5 > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 09-Apr-2015 > Post-History: > Resolution: > > > Abstract > ======== > > This PEP introduces new syntax for coroutines, asynchronous ``with`` > statements and ``for`` loops. The main motivation behind this proposal is to > streamline writing and maintaining asynchronous code, as well as to simplify > previously hard to implement code patterns. > > > Rationale and Goals > =================== > > Current Python supports implementing coroutines via generators (PEP 342), > further enhanced by the ``yield from`` syntax introduced in PEP 380. > This approach has a number of shortcomings: > > * it is easy to confuse coroutines with regular generators, since they share > the same syntax; async libraries often attempt to alleviate this by using > decorators (e.g. ``@asyncio.coroutine`` [1]_); > > * it is not possible to natively define a coroutine which has no ``yield`` > or ``yield from`` statements, again requiring the use of decorators to > fix potential refactoring issues; > > * support for asynchronous calls is limited to expressions where ``yield`` is > allowed syntactically, limiting the usefulness of syntactic features, such > as ``with`` and ``for`` statements. > > This proposal makes coroutines a native Python language feature, and clearly > separates them from generators. This removes generator/coroutine ambiguity, > and makes it possible to reliably define coroutines without reliance on a > specific library. This also enables linters and IDEs to improve static code > analysis and refactoring. > > Native coroutines and the associated new syntax features make it possible > to define context manager and iteration protocols in asynchronous terms. > As shown later in this proposal, the new ``async with`` statement lets Python > programs perform asynchronous calls when entering and exiting a runtime > context, and the new ``async for`` statement makes it possible to perform > asynchronous calls in iterators. > > > Specification > ============= > > This proposal introduces new syntax and semantics to enhance coroutine support > in Python, it does not change the internal implementation of coroutines, which > are still based on generators. > > It is strongly suggested that the reader understands how coroutines are > implemented in Python (PEP 342 and PEP 380). It is also recommended to read > PEP 3156 (asyncio framework). > > From this point in this document we use the word *coroutine* to refer to > functions declared using the new syntax. *generator-based coroutine* is used > where necessary to refer to coroutines that are based on generator syntax. > > > New Coroutine Declaration Syntax > -------------------------------- > > The following new syntax is used to declare a coroutine:: > > async def read_data(db): > pass > > Key properties of coroutines: > > * Coroutines are always generators, even if they do not contain ``await`` > expressions. > > * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in > an ``async`` function. > > * Internally, a new code object flag - ``CO_ASYNC`` - is introduced to enable > runtime detection of coroutines (and migrating existing code). > All coroutines have both ``CO_ASYNC`` and ``CO_GENERATOR`` flags set. > > * Regular generators, when called, return a *generator object*; similarly, > coroutines return a *coroutine object*. > > * ``StopIteration`` exceptions are not propagated out of coroutines, and are > replaced with a ``RuntimeError``. For regular generators such behavior > requires a future import (see PEP 479). > > > types.async_def() > ----------------- > > A new function ``async_def(gen)`` is added to the ``types`` module. It > applies ``CO_ASYNC`` flag to the passed generator's code object, so that it > returns a *coroutine object* when called. > > This feature enables an easy upgrade path for existing libraries. > > > Await Expression > ---------------- > > The following new ``await`` expression is used to obtain a result of coroutine > execution:: > > async def read_data(db): > data = await db.fetch('SELECT ...') > ... > > ``await``, similarly to ``yield from``, suspends execution of ``read_data`` > coroutine until ``db.fetch`` *awaitable* completes and returns the result > data. > > It uses the ``yield from`` implementation with an extra step of validating its > argument. ``await`` only accepts an *awaitable*, which can be one of: > > * A *coroutine object* returned from a coroutine or a generator decorated with > ``types.async_def()``. > > * An object with an ``__await__`` method returning an iterator. > > Any ``yield from`` chain of calls ends with a ``yield``. This is a > fundamental mechanism of how *Futures* are implemented. Since, internally, > coroutines are a special kind of generators, every ``await`` is suspended by > a ``yield`` somewhere down the chain of ``await`` calls (please refer to PEP > 3156 for a detailed explanation.) > > To enable this behavior for coroutines, a new magic method called > ``__await__`` is added. In asyncio, for instance, to enable Future objects > in ``await`` statements, the only change is to add ``__await__ = __iter__`` > line to ``asyncio.Future`` class. > > Objects with ``__await__`` method are called *Future-like* objects in the > rest of this PEP. > > Also, please note that ``__aiter__`` method (see its definition below) cannot > be used for this purpose. It is a different protocol, and would be like > using ``__iter__`` instead of ``__call__`` for regular callables. > > It is a ``SyntaxError`` to use ``await`` outside of a coroutine. > > > Asynchronous Context Managers and "async with" > ---------------------------------------------- > > An *asynchronous context manager* is a context manager that is able to suspend > execution in its *enter* and *exit* methods. > > To make this possible, a new protocol for asynchronous context managers is > proposed. Two new magic methods are added: ``__aenter__`` and ``__aexit__``. > Both must return an *awaitable*. > > An example of an asynchronous context manager:: > > class AsyncContextManager: > async def __aenter__(self): > await log('entering context') > > async def __aexit__(self, exc_type, exc, tb): > await log('exiting context') > > > New Syntax > '''''''''' > > A new statement for asynchronous context managers is proposed:: > > async with EXPR as VAR: > BLOCK > > > which is semantically equivalent to:: > > mgr = (EXPR) > aexit = type(mgr).__aexit__ > aenter = type(mgr).__aenter__(mgr) > exc = True > > try: > try: > VAR = await aenter > BLOCK > except: > exc = False > exit_res = await aexit(mgr, *sys.exc_info()) > if not exit_res: > raise > > finally: > if exc: > await aexit(mgr, None, None, None) > > > As with regular ``with`` statements, it is possible to specify multiple context > managers in a single ``async with`` statement. > > It is an error to pass a regular context manager without ``__aenter__`` and > ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use > ``async with`` outside of a coroutine. > > > Example > ''''''' > > With asynchronous context managers it is easy to implement proper database > transaction managers for coroutines:: > > async def commit(session, data): > ... > > async with session.transaction(): > ... > await session.update(data) > ... > > Code that needs locking also looks lighter:: > > async with lock: > ... > > instead of:: > > with (yield from lock): > ... > > > Asynchronous Iterators and "async for" > -------------------------------------- > > An *asynchronous iterable* is able to call asynchronous code in its *iter* > implementation, and *asynchronous iterator* can call asynchronous code in its > *next* method. To support asynchronous iteration: > > 1. An object must implement an ``__aiter__`` method returning an *awaitable* > resulting in an *asynchronous iterator object*. > > 2. An *asynchronous iterator object* must implement an ``__anext__`` method > returning an *awaitable*. > > 3. To stop iteration```__anext__`` must raise a ``StopAsyncIteration`` > exception. > > An example of asynchronous iterable:: > > class AsyncIterable: > async def __aiter__(self): > return self > > async def __anext__(self): > data = await self.fetch_data() > if data: > return data > else: > raise StopAsyncIteration > > async def fetch_data(self): > ... > > > New Syntax > '''''''''' > > A new statement for iterating through asynchronous iterators is proposed:: > > async for TARGET in ITER: > BLOCK > else: > BLOCK2 > > which is semantically equivalent to:: > > iter = (ITER) > iter = await type(iter).__aiter__(iter) > running = True > while running: > try: > TARGET = await type(iter).__anext__(iter) > except StopAsyncIteration: > running = False > else: > BLOCK > else: > BLOCK2 > > > It is an error to pass a regular iterable without ``__aiter__`` method to > ``async for``. It is a ``SyntaxError`` to use ``async for`` outside of a > coroutine. > > As for with regular ``for`` statement, ``async for`` has an optional ``else`` > clause. > > > Example 1 > ''''''''' > > With asynchronous iteration protocol it is possible to asynchronously buffer > data during iteration:: > > async for data in cursor: > ... > > Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows > of data from a database after every ``N`` iterations. > > The following code illustrates new asynchronous iteration protocol:: > > class Cursor: > def __init__(self): > self.buffer = collections.deque() > > def _prefetch(self): > ... > > async def __aiter__(self): > return self > > async def __anext__(self): > if not self.buffer: > self.buffer = await self._prefetch() > if not self.buffer: > raise StopAsyncIteration > return self.buffer.popleft() > > then the ``Cursor`` class can be used as follows:: > > async for row in Cursor(): > print(row) > > which would be equivalent to the following code:: > > i = await Cursor().__aiter__() > while True: > try: > row = await i.__anext__() > except StopAsyncIteration: > break > else: > print(row) > > > Example 2 > ''''''''' > > The following is a utility class that transforms a regular iterable to an > asynchronous one. While this is not a very useful thing to do, the code > illustrates the relationship between regular and asynchronous iterators. > > :: > > class AsyncIteratorWrapper: > def __init__(self, obj): > self._it = iter(obj) > > async def __aiter__(self): > return self > > async def __anext__(self): > try: > value = next(self._it) > except StopIteration: > raise StopAsyncIteration > return value > > data = "abc" > it = AsyncIteratorWrapper("abc") > async for item in it: > print(it) > > > Why StopAsyncIteration? > ''''''''''''''''''''''' > > Coroutines are still based on generators internally. So, before PEP 479, there > was no fundamental difference between > > :: > > def g1(): > yield from fut > return 'spam' > > and > > :: > > def g2(): > yield from fut > raise StopIteration('spam') > > And since PEP 479 is accepted and enabled by default for coroutines, the > following example will have its ``StopIteration`` wrapped into a > ``RuntimeError`` > > :: > > async def a1(): > await fut > raise StopIteration('spam') > > The only way to tell the outside code that the iteration has ended is to raise > something other than ``StopIteration``. Therefore, a new built-in exception > class ``StopAsyncIteration`` was added. > > Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions raised > in coroutines are wrapped in ``RuntimeError``. > > > Debugging Features > ------------------ > > One of the most frequent mistakes that people make when using generators as > coroutines is forgetting to use ``yield from``:: > > @asyncio.coroutine > def useful(): > asyncio.sleep(1) # this will do noting without 'yield from' > > For debugging this kind of mistakes there is a special debug mode in asyncio, > in which ``@coroutine`` decorator wraps all functions with a special object > with a destructor logging a warning. Whenever a wrapped generator gets garbage > collected, a detailed logging message is generated with information about where > exactly the decorator function was defined, stack trace of where it was > collected, etc. Wrapper object also provides a convenient ``__repr__`` > function with detailed information about the generator. > > The only problem is how to enable these debug capabilities. Since debug > facilities should be a no-op in production mode, ``@coroutine`` decorator makes > the decision of whether to wrap or not to wrap based on an OS environment > variable ``PYTHONASYNCIODEBUG``. This way it is possible to run asyncio > programs with asyncio's own functions instrumented. ``EventLoop.set_debug``, a > different debug facility, has no impact on ``@coroutine`` decorator's behavior. > > With this proposal, coroutines is a native, distinct from generators, > concept. A new method ``set_async_wrapper`` is added to the ``sys`` module, > with which frameworks can provide advanced debugging facilities. > > It is also important to make coroutines as fast and efficient as possible, > therefore there are no debug features enabled by default. > > Example:: > > async def debug_me(): > await asyncio.sleep(1) > > def async_debug_wrap(generator): > return asyncio.AsyncDebugWrapper(generator) > > sys.set_async_wrapper(async_debug_wrap) > > debug_me() # <- this line will likely GC the coroutine object and > # trigger AsyncDebugWrapper's code. > > assert isinstance(debug_me(), AsyncDebugWrapper) > > sys.set_async_wrapper(None) # <- this unsets any previously set wrapper > assert not isinstance(debug_me(), AsyncDebugWrapper) > > If ``sys.set_async_wrapper()`` is called twice, the new wrapper replaces the > previous wrapper. ``sys.set_async_wrapper(None)`` unsets the wrapper. > > > Glossary > ======== > > :Coroutine: > A coroutine function, or just "coroutine", is declared with ``async def``. > It uses ``await`` and ``return value``; see `New Coroutine Declaration > Syntax`_ for details. > > :Coroutine object: > Returned from a coroutine function. See `Await Expression`_ for details. > > :Future-like object: > An object with an ``__await__`` method. It is consumed by ``await`` in a > coroutine. A coroutine waiting for a Future-like object is suspended until > the Future-like object's ``__await__`` completes. ``await`` returns the > result of the Future-like object. See `Await Expression`_ for details. > > :Awaitable: > A *future-like* object or a *coroutine object*. See `Await Expression`_ > for details. > > :Generator-based coroutine: > Coroutines based in generator syntax. Most common example is > ``@asyncio.coroutine``. > > :Asynchronous context manager: > An asynchronous context manager has ``__aenter__`` and ``__aexit__`` methods > and can be used with ``async with``. See > `Asynchronous Context Managers and "async with"`_ for details. > > :Asynchronous iterable: > An object with an ``__aiter__`` method, which must return an *asynchronous > iterator* object. Can be used with ``async for``. See > `Asynchronous Iterators and "async for"`_ for details. > > :Asynchronous iterator: > An asynchronous iterator has an ``__anext__`` method.See > `Asynchronous Iterators and "async for"`_ for details. > > > List of functions and methods > ============================= > > ================= ======================================= ================= > Method Can contain Can't contain > ================= ======================================= ================= > async def func await, return value yield, yield from > async def __a*__ await, return value yield, yield from > def __a*__ return Future-like await > def __await__ yield, yield from, return iterable await > generator yield, yield from, return value await > ================= ======================================= ================= > > Where: > > * ""async def func": coroutine; > > * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, > ``__aexit__`` defined with the ``async`` keyword; > > * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` > defined without the ``async`` keyword, must return an *awaitable*; > > * "def __await__": ``__await__`` method to implement *Future-like* objects; > > * generator: a "regular" generator, function defined with ``def`` and which > contains a least one ``yield`` or ``yield from`` expression. > > *Future-like* is an object with an ``__await__`` method, see > `Await Expression`_ section for details. > > > Transition Plan > =============== > > To avoid backwards compatibility issues with ``async`` and ``await`` keywords, > it was decided to modify ``tokenizer.c`` in such a way, that it: > > * recognizes ``async def`` name tokens combination (start of a coroutine); > > * keeps track of regular functions and coroutines; > > * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with > ``AWAIT`` when in the process of yielding tokens for coroutines. > > This approach allows for seamless combination of new syntax features (all of > them available only in ``async`` functions) with any existing code. > > An example of having "async def" and "async" attribute in one piece of code:: > > class Spam: > async = 42 > > async def ham(): > print(getattr(Spam, 'async')) > > # The coroutine can be executed and will print '42' > > > Backwards Compatibility > ----------------------- > > The only backwards incompatible change is an extra argument ``is_async`` to > ``FunctionDef`` AST node. But since it is a documented fact that the structure > of AST nodes is an implementation detail and subject to change, this should not > be considered a serious issue. > > > Grammar Updates > --------------- > > Grammar changes are also fairly minimal:: > > await_expr: AWAIT test > await_stmt: await_expr > > decorated: decorators (classdef | funcdef | async_funcdef) > async_funcdef: ASYNC funcdef > > async_stmt: ASYNC (funcdef | with_stmt) # will add for_stmt later > > compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt > | funcdef | classdef | decorated | async_stmt) > > atom: ('(' [yield_expr|await_expr|testlist_comp] ')' | > '[' [testlist_comp] ']' | > '{' [dictorsetmaker] '}' | > NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False?) > > expr_stmt: testlist_star_expr (augassign (yield_expr|await_expr|testlist) | > ('=' (yield_expr|await_expr|testlist_star_expr))*) > > > Transition Period Shortcomings > ------------------------------ > > There is just one. > > Until ``async`` and ``await`` are not proper keywords, it is not possible (or > at least very hard) to fix ``tokenizer.c`` to recognize them on the **same > line** with ``def`` keyword:: > > # async and await will always be parsed as variables > > async def outer(): # 1 > def nested(a=(await fut)): > pass > > async def foo(): return (await fut) # 2 > > Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, a > ``SyntaxError`` will be raised. > > To workaround these issues, the above examples can be easily rewritten to a > more readable form:: > > async def outer(): # 1 > a_default = await fut > def nested(a=a_default): > pass > > async def foo(): # 2 > return (await fut) > > This limitation will go away as soon as ``async`` and ``await`` ate proper > keywords. Or if it's decided to use a future import for this PEP. > > > Deprecation Plans > ----------------- > > ``async`` and ``await`` names will be softly deprecated in CPython 3.5 and 3.6. > In 3.7 we will transform them to proper keywords. Making ``async`` and > ``await`` proper keywords before 3.7 might make it harder for people to port > their code to Python 3. > > > asyncio > ------- > > ``asyncio`` module was adapted and tested to work with coroutines and new > statements. Backwards compatibility is 100% preserved. > > The required changes are mainly: > > 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.async_def()`` > function. > > 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. > > 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate > ``async()`` function. > > > Design Considerations > ===================== > > No implicit wrapping in Futures > ------------------------------- > > There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key > difference is that JavaScript "async functions" always return a Promise. While > this approach has some advantages, it also implies that a new Promise object is > created on each "async function" invocation. > > We could implement a similar functionality in Python, by wrapping all > coroutines in a Future object, but this has the following disadvantages: > > 1. Performance. A new Future object would be instantiated on each coroutine > call. Moreover, this makes implementation of ``await`` expressions slower > (disabling optimizations of ``yield from``). > > 2. A new built-in ``Future`` object would need to be added. > > 3. Coming up with a generic ``Future`` interface that is usable for any use > case in any framework is a very hard to solve problem. > > 4. It is not a feature that is used frequently, when most of the code is > coroutines. > > > Why "async" and "await" keywords > -------------------------------- > > async/await is not a new concept in programming languages: > > * C# has it since long time ago [5]_; > > * proposal to add async/await in ECMAScript 7 [2]_; > see also Traceur project [9]_; > > * Facebook's Hack/HHVM [6]_; > > * Google's Dart language [7]_; > > * Scala [8]_; > > * proposal to add async/await to C++ [10]_; > > * and many other less popular languages. > > This is a huge benefit, as some users already have experience with async/await, > and because it makes working with many languages in one project easier (Python > with ECMAScript 7 for instance). > > > Why "__aiter__" is a coroutine > ------------------------------ > > In principle, ``__aiter__`` could be a regular function. There are several > good reasons to make it a coroutine: > > * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` methods are > coroutines, users would often make a mistake defining it as ``async`` > anyways; > > * there might be a need to run some asynchronous operations in ``__aiter__``, > for instance to prepare DB queries or do some file operation. > > > Importance of "async" keyword > ----------------------------- > > While it is possible to just implement ``await`` expression and treat all > functions with at least one ``await`` as coroutines, this approach makes > APIs design, code refactoring and its long time support harder. > > Let's pretend that Python only has ``await`` keyword:: > > def useful(): > ... > await log(...) > ... > > def important(): > await useful() > > If ``useful()`` function is refactored and someone removes all ``await`` > expressions from it, it would become a regular python function, and all code > that depends on it, including ``important()`` would be broken. To mitigate this > issue a decorator similar to ``@asyncio.coroutine`` has to be introduced. > > > Why "async def" > --------------- > > For some people bare ``async name(): pass`` syntax might look more appealing > than ``async def name(): pass``. It is certainly easier to type. But on the > other hand, it breaks the symmetry between ``async def``, ``async with`` and > ``async for``, where ``async`` is a modifier, stating that the statement is > asynchronous. It is also more consistent with the existing grammar. > > > Why not a __future__ import > --------------------------- > > ``__future__`` imports are inconvenient and easy to forget to add. Also, they > are enabled for the whole source file. Consider that there is a big project > with a popular module named "async.py". With future imports it is required to > either import it using ``__import__()`` or ``importlib.import_module()`` calls, > or to rename the module. The proposed approach makes it possible to continue > using old code and modules without a hassle, while coming up with a migration > plan for future python versions. > > > Why magic methods start with "a" > -------------------------------- > > New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, > and ``__aexit__`` all start with the same prefix "a". An alternative proposal > is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. > However, to align new magic methods with the existing ones, such as > ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > > > Why not reuse existing magic names > ---------------------------------- > > An alternative idea about new asynchronous iterators and context managers was > to reuse existing magic methods, by adding an ``async`` keyword to their > declarations:: > > class CM: > async def __enter__(self): # instead of __aenter__ > ... > > This approach has the following downsides: > > * it would not be possible to create an object that works in both ``with`` and > ``async with`` statements; > > * it would look confusing and would require some implicit magic behind the > scenes in the interpreter; > > * one of the main points of this proposal is to make coroutines as simple > and foolproof as possible. > > > Comprehensions > -------------- > > For the sake of restricting the broadness of this PEP there is no new syntax > for asynchronous comprehensions. This should be considered in a separate PEP, > if there is a strong demand for this feature. > > > Performance > =========== > > Overall Impact > -------------- > > This proposal introduces no observable performance impact. Here is an output > of python's official set of benchmarks [4]_: > > :: > > python perf.py -r -b default ../cpython/python.exe ../cpython-aw/python.exe > > [skipped] > > Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: > Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 > x86_64 i386 > > Total CPU cores: 8 > > ### etree_iterparse ### > Min: 0.365359 -> 0.349168: 1.05x faster > Avg: 0.396924 -> 0.379735: 1.05x faster > Significant (t=9.71) > Stddev: 0.01225 -> 0.01277: 1.0423x larger > > The following not significant results are hidden, use -v to show them: > django_v2, 2to3, etree_generate, etree_parse, etree_process, fastpickle, > fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. > > > Tokenizer modifications > ----------------------- > > There is no observable slowdown of parsing python files with the modified > tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated 1000 > times) takes the same amount of time. > > > async/await > ----------- > > The following micro-benchmark was used to determine performance difference > between "async" functions and generators:: > > import sys > import time > > def binary(n): > if n <= 0: > return 1 > l = yield from binary(n - 1) > r = yield from binary(n - 1) > return l + 1 + r > > async def abinary(n): > if n <= 0: > return 1 > l = await abinary(n - 1) > r = await abinary(n - 1) > return l + 1 + r > > def timeit(gen, depth, repeat): > t0 = time.time() > for _ in range(repeat): > list(gen(depth)) > t1 = time.time() > print('{}({}) * {}: total {:.3f}s'.format( > gen.__name__, depth, repeat, t1-t0)) > > The result is that there is no observable performance difference. Minimum > timing of 3 runs > > :: > > abinary(19) * 30: total 12.985s > binary(19) * 30: total 12.953s > > Note that depth of 19 means 1,048,575 calls. > > > Reference Implementation > ======================== > > The reference implementation can be found here: [3]_. > > List of high-level changes and new protocols > -------------------------------------------- > > 1. New syntax for defining coroutines: ``async def`` and new ``await`` > keyword. > > 2. New ``__await__`` method for Future-like objects. > > 3. New syntax for asynchronous context managers: ``async with``. And > associated protocol with ``__aenter__`` and ``__aexit__`` methods. > > 4. New syntax for asynchronous iteration: ``async for``. And associated > protocol with ``__aiter__``, ``__aexit__`` and new built-in exception > ``StopAsyncIteration``. > > 5. New AST nodes: ``AsyncFor``, ``AsyncWith``, ``Await``; ``FunctionDef`` AST > node got a new argument ``is_async``. > > 6. New functions: ``sys.set_async_wrapper(callback)`` and > ``types.async_def(gen)``. > > 7. New ``CO_ASYNC`` bit flag for code objects. > > While the list of changes and new things is not short, it is important to > understand, that most users will not use these features directly. It is > intended to be used in frameworks and libraries to provide users with > convenient to use and unambiguous APIs with ``async def``, ``await``, ``async > for`` and ``async with`` syntax. > > > Working example > --------------- > > All concepts proposed in this PEP are implemented [3]_ and can be tested. > > :: > > import asyncio > > > async def echo_server(): > print('Serving on localhost:8000') > await asyncio.start_server(handle_connection, 'localhost', 8000) > > > async def handle_connection(reader, writer): > print('New connection...') > > while True: > data = await reader.read(8192) > > if not data: > break > > print('Sending {:.10}... back'.format(repr(data))) > writer.write(data) > > > loop = asyncio.get_event_loop() > loop.run_until_complete(echo_server()) > try: > loop.run_forever() > finally: > loop.close() > > > References > ========== > > .. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine > > .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions > > .. [3] https://github.com/1st1/cpython/tree/await > > .. [4] https://hg.python.org/benchmarks > > .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx > > .. [6] http://docs.hhvm.com/manual/en/hack.async.php > > .. [7] https://www.dartlang.org/articles/await-async/ > > .. [8] http://docs.scala-lang.org/sips/pending/async.html > > .. [9] https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental > > .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) > > > Acknowledgments > =============== > > I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew Svetlov, > and ?ukasz Langa for their initial feedback. > > > Copyright > ========= > > This document has been placed in the public domain. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From yselivanov.ml at gmail.com Sat Apr 18 00:18:04 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 18:18:04 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: <5531869C.8090804@gmail.com> Hi Chris, Thanks for the feedback; answers below: On 2015-04-17 6:00 PM, Chris Angelico wrote: > On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov wrote: >> The following new syntax is used to declare a coroutine:: >> >> async def read_data(db): >> pass >> >> Key properties of coroutines: >> >> * Coroutines are always generators, even if they do not contain ``await`` >> expressions. > Do you mean that a coroutine is a special form of generator, or that a > function declared with "async" is always a coroutine even if it > doesn't contain an "await"? You're separating coroutines from > generators in many ways, so maybe I'm just misreading this. Good catch; I've committed the fix. Yes, ``async def`` functions are always coroutines no matter if they have ``await`` exprs in them or not. > >> Asynchronous Context Managers and "async with" >> ---------------------------------------------- >> >> An *asynchronous context manager* is a context manager that is able to >> suspend >> execution in its *enter* and *exit* methods. > A regular 'with' block guarantees that subsequent code won't execute > until __exit__ completes. I presume the same guarantee is applied to > an 'async with' block, and thus it can be used only inside a > coroutine? Correct. > > (Edit: Oops, missed where that's said further down - SyntaxError to > use it outside a coroutine. Might still be worth clarifying the > guarantee somewhere, otherwise ignore me.) > >> New Syntax >> '''''''''' >> >> A new statement for iterating through asynchronous iterators is proposed:: >> >> async for TARGET in ITER: >> BLOCK >> else: >> BLOCK2 >> >> which is semantically equivalent to:: >> >> iter = (ITER) >> iter = await type(iter).__aiter__(iter) >> running = True >> while running: >> try: >> TARGET = await type(iter).__anext__(iter) >> except StopAsyncIteration: >> running = False >> else: >> BLOCK >> else: >> BLOCK2 > (Not sure why you don't just use "break" instead of "running = False"? > Maybe I'm blind to some distinction here.) Because if you use break, the "else: BLOCK2" won't execute, but it should, as StopAsyncIteration is not a regular exception, but something to signal that the iteration is over. > >> Why StopAsyncIteration? >> ''''''''''''''''''''''' >> >> Coroutines are still based on generators internally. So, before PEP 479, >> there >> was no fundamental difference between >> >> :: >> >> def g1(): >> yield from fut >> return 'spam' >> >> and >> >> :: >> >> def g2(): >> yield from fut >> raise StopIteration('spam') >> >> And since PEP 479 is accepted and enabled by default for coroutines, the >> following example will have its ``StopIteration`` wrapped into a >> ``RuntimeError`` >> >> :: >> >> async def a1(): >> await fut >> raise StopIteration('spam') >> >> The only way to tell the outside code that the iteration has ended is to >> raise >> something other than ``StopIteration``. Therefore, a new built-in exception >> class ``StopAsyncIteration`` was added. >> >> Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions >> raised >> in coroutines are wrapped in ``RuntimeError``. > Can't a coroutine simply use 'return'? I'm not hugely familiar with > them, but I'm not understanding here why you need a completely > separate exception - and, more importantly, why you wouldn't have the > same consideration of "StopAsyncIteration leakage". It ought to be > possible for the coroutine to return, StopIteration be synthesized, > and the __anext__ function to bubble that out - exactly the way it > does for generators today. Unfortunately, there is no way we can enable StopIteration to do the same thing that StopAsyncIteration does. 'return' in coroutines is implemented with 'StopIteration' exception internally, so there is no way (without ugly hacks) to distinguish before plain returns and desire to stop the iteration. > >> Glossary >> ======== >> >> :Coroutine: >> A coroutine function, or just "coroutine", is declared with ``async >> def``. >> It uses ``await`` and ``return value``; see `New Coroutine Declaration >> Syntax`_ for details. > (Using this definition to justify my statement above) > >> Why not a __future__ import >> --------------------------- >> >> ``__future__`` imports are inconvenient and easy to forget to add. Also, >> they >> are enabled for the whole source file. Consider that there is a big project >> with a popular module named "async.py". With future imports it is required >> to >> either import it using ``__import__()`` or ``importlib.import_module()`` >> calls, >> or to rename the module. The proposed approach makes it possible to >> continue >> using old code and modules without a hassle, while coming up with a >> migration >> plan for future python versions. > The clash would occur only in modules that use the __future__ import, > wouldn't it? And those modules are going to have to be fixed by > version X.Y anyway - the whole point of the __future__ import is to > detect that. You're right. Let's see what python-ideas thinks about it. I'm fine if everybody wants __future__ imports. I will only have to rollback my changes in tokenizer.c and change few tokens in Grammar to make the reference implementation work. >> Why magic methods start with "a" >> -------------------------------- >> >> New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, >> and ``__aexit__`` all start with the same prefix "a". An alternative >> proposal >> is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. >> However, to align new magic methods with the existing ones, such as >> ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > I support the short names, with the possible exception that > "__await__" now looks like "asynchronous wait", which may or may not > be desirable. > >> Comprehensions >> -------------- >> >> For the sake of restricting the broadness of this PEP there is no new syntax >> for asynchronous comprehensions. This should be considered in a separate >> PEP, >> if there is a strong demand for this feature. > Likewise asynchronous lambda functions (or maybe I just missed it). Right. Right now I'm just not sure they would be useful at all, whereas I can imagine some usecases for comprehensions ;) > > > Overall comment: The changes appear to be infecting a lot of code. > Will this suddenly be something that everyone has to think about, or > can it still be ignored in projects that don't use it? If you're not writing some asynchronous code or using some asynchronous library that takes advantage of coroutines, you can completely ignore the new syntax and protocols. > There'll be a > few object types that would benefit from becoming async context > managers as well as regular context managers (I would expect, for > instance, that the built-in file type would allow its auto-closing to > be asynchronous), but if a third-party module completely ignores this > proposal, everything should continue to work, right? Exactly. That's why we propose to have separate protocols __a*__ to completely eliminate possibility of screwing something up. > > I'm seeing the potential for a lot of code duplication here, but > possibly once things settle into real-world usage, a few easy > decorators will help out with that (like the total_ordering decorator > for handling __gt__, __le__, etc etc). > > +1 on the proposal as a whole. Awesome! Thanks a lot! Yury From guido at python.org Sat Apr 18 00:19:41 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 15:19:41 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> Message-ID: So propose a patch for callable() along these lines. It'll have to be C code. On Fri, Apr 17, 2015 at 2:21 PM, Ionel Cristian M?rie? wrote: > Yes indeed, that's one way but I wouldn't want to monkeypatch the > `callable` builtin. People wouldn't expect that library would even dare > monkeypatch builtin. > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman wrote: > >> On 04/18, Ionel Cristian M?rie? wrote: >> >> > Also there's the issue about not being able to implement a true proxy >> (as >> > outlined before). >> >> Proxies are a bit of a pain. But you can create your own callable >> function. >> >> Something like (untested): >> >> def callable(obj): >> try: >> func = obj.__call__ >> return True >> except AttributeError: >> return False >> >> -- >> ~Ethan~ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Apr 18 00:20:48 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 17 Apr 2015 15:20:48 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: On Apr 17, 2015, at 15:00, Chris Angelico wrote: > >> On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov wrote: >> The following new syntax is used to declare a coroutine:: >> >> async def read_data(db): >> pass >> >> Key properties of coroutines: >> >> * Coroutines are always generators, even if they do not contain ``await`` >> expressions. > > Do you mean that a coroutine is a special form of generator, or that a > function declared with "async" is always a coroutine even if it > doesn't contain an "await"? You're separating coroutines from > generators in many ways, so maybe I'm just misreading this. > >> Asynchronous Context Managers and "async with" >> ---------------------------------------------- >> >> An *asynchronous context manager* is a context manager that is able to >> suspend >> execution in its *enter* and *exit* methods. > > A regular 'with' block guarantees that subsequent code won't execute > until __exit__ completes. I presume the same guarantee is applied to > an 'async with' block, and thus it can be used only inside a > coroutine? > > (Edit: Oops, missed where that's said further down - SyntaxError to > use it outside a coroutine. Might still be worth clarifying the > guarantee somewhere, otherwise ignore me.) > >> New Syntax >> '''''''''' >> >> A new statement for iterating through asynchronous iterators is proposed:: >> >> async for TARGET in ITER: >> BLOCK >> else: >> BLOCK2 >> >> which is semantically equivalent to:: >> >> iter = (ITER) >> iter = await type(iter).__aiter__(iter) >> running = True >> while running: >> try: >> TARGET = await type(iter).__anext__(iter) >> except StopAsyncIteration: >> running = False >> else: >> BLOCK >> else: >> BLOCK2 > > (Not sure why you don't just use "break" instead of "running = False"? > Maybe I'm blind to some distinction here.) break in a loop triggers the else; failing the condition doesn't. So that change would mean BLOCK2 always runs, instead of only running if there's an explicit break in BLOCK. > >> Why StopAsyncIteration? >> ''''''''''''''''''''''' >> >> Coroutines are still based on generators internally. So, before PEP 479, >> there >> was no fundamental difference between >> >> :: >> >> def g1(): >> yield from fut >> return 'spam' >> >> and >> >> :: >> >> def g2(): >> yield from fut >> raise StopIteration('spam') >> >> And since PEP 479 is accepted and enabled by default for coroutines, the >> following example will have its ``StopIteration`` wrapped into a >> ``RuntimeError`` >> >> :: >> >> async def a1(): >> await fut >> raise StopIteration('spam') >> >> The only way to tell the outside code that the iteration has ended is to >> raise >> something other than ``StopIteration``. Therefore, a new built-in exception >> class ``StopAsyncIteration`` was added. >> >> Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions >> raised >> in coroutines are wrapped in ``RuntimeError``. > > Can't a coroutine simply use 'return'? I'm not hugely familiar with > them, but I'm not understanding here why you need a completely > separate exception - and, more importantly, why you wouldn't have the > same consideration of "StopAsyncIteration leakage". It ought to be > possible for the coroutine to return, StopIteration be synthesized, > and the __anext__ function to bubble that out - exactly the way it > does for generators today. > >> Glossary >> ======== >> >> :Coroutine: >> A coroutine function, or just "coroutine", is declared with ``async >> def``. >> It uses ``await`` and ``return value``; see `New Coroutine Declaration >> Syntax`_ for details. > > (Using this definition to justify my statement above) > >> Why not a __future__ import >> --------------------------- >> >> ``__future__`` imports are inconvenient and easy to forget to add. Also, >> they >> are enabled for the whole source file. Consider that there is a big project >> with a popular module named "async.py". With future imports it is required >> to >> either import it using ``__import__()`` or ``importlib.import_module()`` >> calls, >> or to rename the module. The proposed approach makes it possible to >> continue >> using old code and modules without a hassle, while coming up with a >> migration >> plan for future python versions. > > The clash would occur only in modules that use the __future__ import, > wouldn't it? And those modules are going to have to be fixed by > version X.Y anyway - the whole point of the __future__ import is to > detect that. > >> Why magic methods start with "a" >> -------------------------------- >> >> New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, >> and ``__aexit__`` all start with the same prefix "a". An alternative >> proposal >> is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. >> However, to align new magic methods with the existing ones, such as >> ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > > I support the short names, with the possible exception that > "__await__" now looks like "asynchronous wait", which may or may not > be desirable. > >> Comprehensions >> -------------- >> >> For the sake of restricting the broadness of this PEP there is no new syntax >> for asynchronous comprehensions. This should be considered in a separate >> PEP, >> if there is a strong demand for this feature. > > Likewise asynchronous lambda functions (or maybe I just missed it). Do you think those would be useful? Anything small enough to be worth writing inline seems like it would be swamped by almost doubling the syntactic overhead, and therefore a lot less likely to still be worth writing inline... But maybe there's still a wide enough range where that wouldn't be true? > Overall comment: The changes appear to be infecting a lot of code. > Will this suddenly be something that everyone has to think about, or > can it still be ignored in projects that don't use it? There'll be a > few object types that would benefit from becoming async context > managers as well as regular context managers (I would expect, for > instance, that the built-in file type would allow its auto-closing to > be asynchronous), but if a third-party module completely ignores this > proposal, everything should continue to work, right? > > I'm seeing the potential for a lot of code duplication here, but > possibly once things settle into real-world usage, a few easy > decorators will help out with that (like the total_ordering decorator > for handling __gt__, __le__, etc etc). > > +1 on the proposal as a whole. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From yselivanov.ml at gmail.com Sat Apr 18 00:22:39 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 18:22:39 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: <553187AF.9050001@gmail.com> On 2015-04-17 6:20 PM, Andrew Barnert wrote: > On Apr 17, 2015, at 15:00, Chris Angelico wrote: [...] >>> Comprehensions >>> -------------- >>> >>> For the sake of restricting the broadness of this PEP there is no new syntax >>> for asynchronous comprehensions. This should be considered in a separate >>> PEP, >>> if there is a strong demand for this feature. >> Likewise asynchronous lambda functions (or maybe I just missed it). > Do you think those would be useful? Anything small enough to be worth writing inline seems like it would be swamped by almost doubling the syntactic overhead, and therefore a lot less likely to still be worth writing inline... But maybe there's still a wide enough range where that wouldn't be true? To be honest, I think that typing 'async lambda' doesn't make things look small ;) Yury From yselivanov.ml at gmail.com Sat Apr 18 00:25:25 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 18:25:25 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: <55318855.7040905@gmail.com> On 2015-04-17 6:15 PM, Andrew Barnert wrote: > Could you show the best yield from parallel for the example at the end, so it's easier to see the readability benefits? > > Or, even better, provide a simple example that uses async for or async with and the yield from parallel, so it's easier to see the (I'm assuming even bigger, by a long shot) readability benefits? I don't have a real example, but we're getting there. Andrew Svetlov made some experiments porting his aiopg library to the new syntax. So perhaps early next week we will have a real example of real code. You can also experiment yourself with the implementation. Thanks you, Yury From abarnert at yahoo.com Sat Apr 18 00:26:52 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 17 Apr 2015 15:26:52 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <5531869C.8090804@gmail.com> References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> Message-ID: <3D2DAC8B-7251-42DC-AAE0-C4987277ED86@yahoo.com> On Apr 17, 2015, at 15:18, Yury Selivanov wrote: > > Hi Chris, > > Thanks for the feedback; answers below: > >> On 2015-04-17 6:00 PM, Chris Angelico wrote: >>> On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov wrote: >>> The following new syntax is used to declare a coroutine:: >>> >>> async def read_data(db): >>> pass >>> >>> Key properties of coroutines: >>> >>> * Coroutines are always generators, even if they do not contain ``await`` >>> expressions. >> Do you mean that a coroutine is a special form of generator, or that a >> function declared with "async" is always a coroutine even if it >> doesn't contain an "await"? You're separating coroutines from >> generators in many ways, so maybe I'm just misreading this. > > Good catch; I've committed the fix. > > Yes, ``async def`` functions are always coroutines no matter if > they have ``await`` exprs in them or not. >> >>> Asynchronous Context Managers and "async with" >>> ---------------------------------------------- >>> >>> An *asynchronous context manager* is a context manager that is able to >>> suspend >>> execution in its *enter* and *exit* methods. >> A regular 'with' block guarantees that subsequent code won't execute >> until __exit__ completes. I presume the same guarantee is applied to >> an 'async with' block, and thus it can be used only inside a >> coroutine? > Correct. >> >> (Edit: Oops, missed where that's said further down - SyntaxError to >> use it outside a coroutine. Might still be worth clarifying the >> guarantee somewhere, otherwise ignore me.) >> >>> New Syntax >>> '''''''''' >>> >>> A new statement for iterating through asynchronous iterators is proposed:: >>> >>> async for TARGET in ITER: >>> BLOCK >>> else: >>> BLOCK2 >>> >>> which is semantically equivalent to:: >>> >>> iter = (ITER) >>> iter = await type(iter).__aiter__(iter) >>> running = True >>> while running: >>> try: >>> TARGET = await type(iter).__anext__(iter) >>> except StopAsyncIteration: >>> running = False >>> else: >>> BLOCK >>> else: >>> BLOCK2 >> (Not sure why you don't just use "break" instead of "running = False"? >> Maybe I'm blind to some distinction here.) > > Because if you use break, the "else: BLOCK2" won't execute, but > it should, as StopAsyncIteration is not a regular exception, but > something to signal that the iteration is over. >> >>> Why StopAsyncIteration? >>> ''''''''''''''''''''''' >>> >>> Coroutines are still based on generators internally. So, before PEP 479, >>> there >>> was no fundamental difference between >>> >>> :: >>> >>> def g1(): >>> yield from fut >>> return 'spam' >>> >>> and >>> >>> :: >>> >>> def g2(): >>> yield from fut >>> raise StopIteration('spam') >>> >>> And since PEP 479 is accepted and enabled by default for coroutines, the >>> following example will have its ``StopIteration`` wrapped into a >>> ``RuntimeError`` >>> >>> :: >>> >>> async def a1(): >>> await fut >>> raise StopIteration('spam') >>> >>> The only way to tell the outside code that the iteration has ended is to >>> raise >>> something other than ``StopIteration``. Therefore, a new built-in exception >>> class ``StopAsyncIteration`` was added. >>> >>> Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions >>> raised >>> in coroutines are wrapped in ``RuntimeError``. >> Can't a coroutine simply use 'return'? I'm not hugely familiar with >> them, but I'm not understanding here why you need a completely >> separate exception - and, more importantly, why you wouldn't have the >> same consideration of "StopAsyncIteration leakage". It ought to be >> possible for the coroutine to return, StopIteration be synthesized, >> and the __anext__ function to bubble that out - exactly the way it >> does for generators today. > > Unfortunately, there is no way we can enable StopIteration to do > the same thing that StopAsyncIteration does. > > 'return' in coroutines is implemented with 'StopIteration' exception > internally, so there is no way (without ugly hacks) to distinguish > before plain returns and desire to stop the iteration. >> >>> Glossary >>> ======== >>> >>> :Coroutine: >>> A coroutine function, or just "coroutine", is declared with ``async >>> def``. >>> It uses ``await`` and ``return value``; see `New Coroutine Declaration >>> Syntax`_ for details. >> (Using this definition to justify my statement above) >> >>> Why not a __future__ import >>> --------------------------- >>> >>> ``__future__`` imports are inconvenient and easy to forget to add. Also, >>> they >>> are enabled for the whole source file. Consider that there is a big project >>> with a popular module named "async.py". With future imports it is required >>> to >>> either import it using ``__import__()`` or ``importlib.import_module()`` >>> calls, >>> or to rename the module. The proposed approach makes it possible to >>> continue >>> using old code and modules without a hassle, while coming up with a >>> migration >>> plan for future python versions. >> The clash would occur only in modules that use the __future__ import, >> wouldn't it? And those modules are going to have to be fixed by >> version X.Y anyway - the whole point of the __future__ import is to >> detect that. > You're right. > > Let's see what python-ideas thinks about it. I'm fine if everybody > wants __future__ imports. I will only have to rollback my changes > in tokenizer.c and change few tokens in Grammar to make the > reference implementation work. >>> Why magic methods start with "a" >>> -------------------------------- >>> >>> New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, >>> and ``__aexit__`` all start with the same prefix "a". An alternative >>> proposal >>> is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. >>> However, to align new magic methods with the existing ones, such as >>> ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. >> I support the short names, with the possible exception that >> "__await__" now looks like "asynchronous wait", which may or may not >> be desirable. >> >>> Comprehensions >>> -------------- >>> >>> For the sake of restricting the broadness of this PEP there is no new syntax >>> for asynchronous comprehensions. This should be considered in a separate >>> PEP, >>> if there is a strong demand for this feature. >> Likewise asynchronous lambda functions (or maybe I just missed it). > Right. Right now I'm just not sure they would be useful at all, > whereas I can imagine some usecases for comprehensions ;) > >> >> >> Overall comment: The changes appear to be infecting a lot of code. >> Will this suddenly be something that everyone has to think about, or >> can it still be ignored in projects that don't use it? > > If you're not writing some asynchronous code or using some > asynchronous library that takes advantage of coroutines, you > can completely ignore the new syntax and protocols. Is that true? If I'm writing a general-purpose library, and it might be useful in someone else's async code, and I write a context manager that does the usual "close/destroy resources on __exit__ thing", shouldn't I really think about whether it should also be an async context manager? That raises another question: is there a reasonable parallel to the contextlib.contextmanager decorator for building simple async context managers? (Or, for that matter, for building simple context-manager-and-also-async-context-managers?) > >> There'll be a >> few object types that would benefit from becoming async context >> managers as well as regular context managers (I would expect, for >> instance, that the built-in file type would allow its auto-closing to >> be asynchronous), but if a third-party module completely ignores this >> proposal, everything should continue to work, right? > Exactly. That's why we propose to have separate protocols > __a*__ to completely eliminate possibility of screwing > something up. > >> >> I'm seeing the potential for a lot of code duplication here, but >> possibly once things settle into real-world usage, a few easy >> decorators will help out with that (like the total_ordering decorator >> for handling __gt__, __le__, etc etc). >> >> +1 on the proposal as a whole. > Awesome! > > Thanks a lot! > > Yury > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From dw+python-ideas at hmmz.org Sat Apr 18 00:24:10 2015 From: dw+python-ideas at hmmz.org (David Wilson) Date: Fri, 17 Apr 2015 22:24:10 +0000 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: <20150417222410.GA25999@k3> On Fri, Apr 17, 2015 at 02:58:58PM -0400, Yury Selivanov wrote: > I believe that PEPs 380 and 3156 were a major breakthrough for Python > 3, and I hope that this PEP can be another big step. Who knows, maybe > it will be one of the reasons to drive people's interest towards > Python 3. An offer like this would assuredly and irrevocably propel me from my "Python 3 conservative" position and well into the "Python 3 zealot" stratosphere :) A great deal of thought has obviously gone into your proposal, so I expect you already have answers to these: Regarding introduction of async/await keywords, could e.g. the tokenizer change be avoided by moving the 'async' keyword after the parameter list? like "def foo(x) async:". That would seem parseable without introducing any new token, since the parser could be made to error out if any identifier except "async" appears, similarly instead of introducing an 'await' keyword, 'yield await' might be possible. (Please forgive me if my parsing ignorance is showing through) Further regarding the introduction of the await keyword, within an async function did you consider the possibility of making suspension of the coroutine implicit? >From the perspective of the interpreter, objects exposing __await__ might grow a .tp_await slot which could trivially be tested for NULL during e.g. the CALL_FUNCTION opcode epilog, though I guess this stuff strays quite far from well understod async/await designs from other languages (e.g. C#). I quite like the idea of folding the mechanism out of the user's sight as far as practical, though I don't know if that's possible without introducing ambiguities. Is there a need or desire for "async lambda ...:"? Notably, it's possible though fairly useless to define a lambda generator: "lambda x: (yield from x)" works today. Generally the magic to avoid backwards compatibility issues seem less desirable than simply including a new __future__ mode, though the stated reasons for the trickery are good. Somewhat contradicting my last point, __future__ serves as very useful documentation for a file's implementation. Regarding the problematic module import example, that might easily be solved by simple file rename and introducing a stub module for old code like: async_impl.py: # Guts of old async.py async.py: from .async_impl import * # NOQA Similarly for object attributes and suchlike, properties might suffice. Thanks for proposing this! Off to buypopcorn, very much looking forward to seeing how the thread plays out. ;) David From contact at ionelmc.ro Sat Apr 18 00:32:34 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 01:32:34 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: Yes, it could test first for obj->ob_type->tp_call - no point in testing the instance if the type doesn't have it at all. Regarding the property and how it's used for magic methods, as I understood the cpython code, the slots on the type hold some special wrappers (PyWrapperDescrObject?) that respect the descriptor protocol when called from C (this what makes staticmethod, classmethod and incidentally, property and descriptors in general work for special methods). Eg: https://hg.python.org/cpython/file/v3.4.2/Objects/typeobject.c#l6402 To me all that looks very intentional. It's just that it's not consistent. But what do I know, I've just read the code :-) ? So all I?'m asking is to add an extra check for hasattr(x, "__call__") in the callable builtin. It shouldn't break any code, and it would make `callable` a little bit more reliable. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 12:39 AM, Guido van Rossum wrote: > I think you've found an unintended and undocumented backdoor. I admit I > don't understand how this works in CPython. Overloaded operators like > __add__ or __call__ should be methods in the class, and we don't look for > them in the instance. But somehow defining them with @property works (I > guess because @property is in the class). > > What's different for __call__ is that callable() exists. And this is > probably why I exorcised it Python 3.0 -- but apparently it's back. :-( > > In the end callable() doesn't always produce a correct answer; but maybe > we can make it work in this case by first testing the class and then the > instance? Something like (untested): > > def callable(x): > return hasattr(x.__class__, '__call__') and hasattr(x, '__call__') > > > On Fri, Apr 17, 2015 at 2:19 PM, Ionel Cristian M?rie? > wrote: > >> __add__ as a property/descriptor seems to work fine, eg: >> >> >>> class C: >>> ... @property >>> ... def __add__(self): >>> ... return lambda other: [self, other] >>> ... >>> >>> C() + C() >>> [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at >>> 0x0000000003652CC0>] >>> >> >> Am I missing something? >> >> >> Thanks, >> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >> >> On Sat, Apr 18, 2015 at 12:15 AM, Guido van Rossum >> wrote: >> >>> You won't have any more luck defining __add__ as a property -- just >>> don't do that. >>> >>> On how to implement a proxy, I'll let other explain. But this is not it. >>> >>> On Fri, Apr 17, 2015 at 2:04 PM, Ionel Cristian M?rie? < >>> contact at ionelmc.ro> wrote: >>> >>>> Well yes, from that example it look right, because the call operator >>>> uses the __call__ attribute from the type of the object. However, when the >>>> call operator gets the __call__ method it will actually use it as a >>>> descriptor. From that perspective it's inconsistent. >>>> >>>> Also there's the issue about not being able to implement a true proxy >>>> (as outlined before). >>>> >>>> What actually prevents this being fixed? >>>> >>>> >>>> Thanks, >>>> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >>>> >>>> On Fri, Apr 17, 2015 at 11:55 PM, Guido van Rossum >>>> wrote: >>>> >>>>> I think you're fighting windmills. Like most special operations (e.g. >>>>> __add__), a __call__ attribute on the object does not work, i.e. it does >>>>> not make the object callable. E.g. >>>>> >>>>> $ python3 >>>>> Python 3.5.0a2 (v3.5.0a2:0337bd7ebcb6, Mar 8 2015, 01:12:06) >>>>> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin >>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>> >>> class C: pass >>>>> ... >>>>> >>> c = C() >>>>> >>> c.__call__ = lambda *a: a >>>>> >>> c() >>>>> Traceback (most recent call last): >>>>> File "", line 1, in >>>>> TypeError: 'C' object is not callable >>>>> >>> callable(c) >>>>> False >>>>> >>> hasattr(c, '__call__') >>>>> True >>>>> >>> >>>>> >>>>> On Fri, Apr 17, 2015 at 1:45 PM, Ionel Cristian M?rie? < >>>>> contact at ionelmc.ro> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I had an issue today with the `callable` builtin because it doesn't >>>>>> correctly check that the object has the __call__ attribute. >>>>>> >>>>>> Effectively what `callable(a)` does is `hasattr(type(a), '__call__')` >>>>>> but that's not very straightforward. A more straightforward implementation >>>>>> would do something like `hasattr(a, '__call__')`. >>>>>> >>>>>> For example: >>>>>> >>>>>> Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 >>>>>>> 64 bit (AMD64)] on win32 >>>>>>> Type "help", "copyright", "credits" or "license" for more >>>>>>> information. >>>>>>> >>> callable >>>>>>> >>>>>>> >>> class A: >>>>>>> ... @property >>>>>>> ... def __call__(self): >>>>>>> ... raise AttributeError('go away') >>>>>>> ... >>>>>>> >>> a = A() >>>>>>> >>> a >>>>>>> <__main__.A object at 0x000000000365B5C0> >>>>>>> >>> a.__call__ >>>>>>> Traceback (most recent call last): >>>>>>> File "", line 1, in >>>>>>> File "", line 4, in __call__ >>>>>>> AttributeError: go away >>>>>>> >>> callable(a) >>>>>>> True >>>>>>> >>> # it should be False :( >>>>>>> >>>>>> >>>>>> ?So it boils down to this: >>>>>> >>>>>>> >>> hasattr(a, "__call__") >>>>>>> False >>>>>>> >>> hasattr(type(a), "__call__") >>>>>>> True >>>>>> >>>>>> >>>>>> My issue is that I didn't call `callable(type(a))` but just >>>>>> `callable(a)`. Clearly mismatching what happens when you do hasattr(a, >>>>>> "__call__"). >>>>>> >>>>>> To put in contrast, this is legal and clearly indicates the >>>>>> descriptors are being used as expected: >>>>>> >>>>>> >>> class B: >>>>>>> ... @property >>>>>>> ... def __call__(self): >>>>>>> ... return lambda: 1 >>>>>>> ... >>>>>>> >>> b = B() >>>>>>> >>> b() >>>>>>> 1 >>>>>>> >>>>>> >>>>>> ?There? >>>>>> ?'s some more discussing in issue 23990 >>>>>> ? >>>>>> ? where I get slightly angry, sorry.? >>>>>> >>>>>> >>>>>> ?So were is this change actually useful? Proxies! Since new-style >>>>>> objects in Python you cannot really proxy the callable aspect of objects, >>>>>> because `callable` just checks that a field is set in a C struct.? >>>>>> ? This is fairly inconvenient because you have to know upfront if >>>>>> your target is going to be callable or not.? >>>>>> >>>>>> >>>>>> >>>>>> Thanks, >>>>>> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >>>>>> >>>>>> _______________________________________________ >>>>>> Python-ideas mailing list >>>>>> Python-ideas at python.org >>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> --Guido van Rossum (python.org/~guido) >>>>> >>>> >>>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Sat Apr 18 00:34:14 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 01:34:14 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> Message-ID: Ah, yes, someone else has already made a patch here: http://bugs.python.org/file39090/callable.diff This is the issue: http://bugs.python.org/issue23990 Would you add your thoughts there? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Sat, Apr 18, 2015 at 1:19 AM, Guido van Rossum wrote: > So propose a patch for callable() along these lines. It'll have to be C > code. > > On Fri, Apr 17, 2015 at 2:21 PM, Ionel Cristian M?rie? > wrote: > >> Yes indeed, that's one way but I wouldn't want to monkeypatch the >> `callable` builtin. People wouldn't expect that library would even dare >> monkeypatch builtin. >> >> >> Thanks, >> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >> >> On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman >> wrote: >> >>> On 04/18, Ionel Cristian M?rie? wrote: >>> >>> > Also there's the issue about not being able to implement a true proxy >>> (as >>> > outlined before). >>> >>> Proxies are a bit of a pain. But you can create your own callable >>> function. >>> >>> Something like (untested): >>> >>> def callable(obj): >>> try: >>> func = obj.__call__ >>> return True >>> except AttributeError: >>> return False >>> >>> -- >>> ~Ethan~ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Sat Apr 18 00:36:48 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 18:36:48 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150417222410.GA25999@k3> References: <553157F2.8060408@gmail.com> <20150417222410.GA25999@k3> Message-ID: <55318B00.9080805@gmail.com> Hi David, Thanks a lot for feedback; answers below: On 2015-04-17 6:24 PM, David Wilson wrote: > On Fri, Apr 17, 2015 at 02:58:58PM -0400, Yury Selivanov wrote: > >> I believe that PEPs 380 and 3156 were a major breakthrough for Python >> 3, and I hope that this PEP can be another big step. Who knows, maybe >> it will be one of the reasons to drive people's interest towards >> Python 3. > An offer like this would assuredly and irrevocably propel me from my > "Python 3 conservative" position and well into the "Python 3 zealot" > stratosphere :) Such a relief to hear that! > > A great deal of thought has obviously gone into your proposal, so I > expect you already have answers to these: > > > Regarding introduction of async/await keywords, could e.g. the > tokenizer change be avoided by moving the 'async' keyword after the > parameter list? like "def foo(x) async:". That would seem parseable > without introducing any new token, since the parser could be made to > error out if any identifier except "async" appears, similarly instead of > introducing an 'await' keyword, 'yield await' might be possible. (Please > forgive me if my parsing ignorance is showing through) I'm not entirely sure that it's possible to modify tokenizer to understand "def() async:" For instance, for the this code: "async = 1; print = 2; lst[async:print]" we'll need to do a hell of a look-ahead. Or do some smart stack tracking.. Better just to either use the proposed mechanism or __future__ import. > > > Further regarding the introduction of the await keyword, within an > async function did you consider the possibility of making suspension of > the coroutine implicit? That would imply implementing mechanisms similar to what greenlets module does, or stackless python. My understanding is that is never going to happen, for better or worse. > > From the perspective of the interpreter, objects exposing __await__ > might grow a .tp_await slot which could trivially be tested for NULL > during e.g. the CALL_FUNCTION opcode epilog, though I guess this stuff > strays quite far from well understod async/await designs from other > languages (e.g. C#). I think this is an optimization (or better C api?) that we can consider later. > > I quite like the idea of folding the mechanism out of the user's sight > as far as practical, though I don't know if that's possible without > introducing ambiguities. > > > Is there a need or desire for "async lambda ...:"? Notably, it's > possible though fairly useless to define a lambda generator: > "lambda x: (yield from x)" works today. If we use __future__ import it should be possible to modify grammar to parse that. Although I don't like the syntax. > > > Generally the magic to avoid backwards compatibility issues seem > less desirable than simply including a new __future__ mode, though the > stated reasons for the trickery are good. Somewhat contradicting my last > point, __future__ serves as very useful documentation for a file's > implementation. I'm not opposed to have __future__ imports. If most people think that they are better, we'll update the proposal and reference implementation. > > Regarding the problematic module import example, that might easily be > solved by simple file rename and introducing a stub module for old code > like: > > async_impl.py: > # Guts of old async.py > > async.py: > from .async_impl import * # NOQA > > Similarly for object attributes and suchlike, properties might suffice. Agree! > > Thanks for proposing this! Off to buypopcorn, very much looking forward > to seeing how the thread plays out. ;) Thanks a lot again! Yury From yselivanov.ml at gmail.com Sat Apr 18 00:40:01 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 18:40:01 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <3D2DAC8B-7251-42DC-AAE0-C4987277ED86@yahoo.com> References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> <3D2DAC8B-7251-42DC-AAE0-C4987277ED86@yahoo.com> Message-ID: <55318BC1.6060203@gmail.com> Andrew, On 2015-04-17 6:26 PM, Andrew Barnert wrote: > On Apr 17, 2015, at 15:18, Yury Selivanov wrote: >> Hi Chris, [...] >> >>> >>> Overall comment: The changes appear to be infecting a lot of code. >>> Will this suddenly be something that everyone has to think about, or >>> can it still be ignored in projects that don't use it? >> If you're not writing some asynchronous code or using some >> asynchronous library that takes advantage of coroutines, you >> can completely ignore the new syntax and protocols. > Is that true? > > If I'm writing a general-purpose library, and it might be useful in someone else's async code, and I write a context manager that does the usual "close/destroy resources on __exit__ thing", shouldn't I really think about whether it should also be an async context manager? You can still use regular 'with' and 'for' statements in your async code. If someone by accident tries to use your regular context manager with 'async with' she'll get an exception. > > That raises another question: is there a reasonable parallel to the contextlib.contextmanager decorator for building simple async context managers? (Or, for that matter, for building simple context-manager-and-also-async-context-managers?) No, I don't think that we can have something similar to 'contextmanager' decorator. That would require some kind of a 'metayield' keyword ;) Thanks! Yury From guido at python.org Sat Apr 18 00:38:15 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 15:38:15 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> Message-ID: No, but you can refer to this thread. On Fri, Apr 17, 2015 at 3:34 PM, Ionel Cristian M?rie? wrote: > Ah, yes, someone else has already made a patch here: > http://bugs.python.org/file39090/callable.diff > > This is the issue: http://bugs.python.org/issue23990 > > Would you add your thoughts there? > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > > On Sat, Apr 18, 2015 at 1:19 AM, Guido van Rossum > wrote: > >> So propose a patch for callable() along these lines. It'll have to be C >> code. >> >> On Fri, Apr 17, 2015 at 2:21 PM, Ionel Cristian M?rie? < >> contact at ionelmc.ro> wrote: >> >>> Yes indeed, that's one way but I wouldn't want to monkeypatch the >>> `callable` builtin. People wouldn't expect that library would even dare >>> monkeypatch builtin. >>> >>> >>> Thanks, >>> -- Ionel Cristian M?rie?, http://blog.ionelmc.ro >>> >>> On Sat, Apr 18, 2015 at 12:10 AM, Ethan Furman >>> wrote: >>> >>>> On 04/18, Ionel Cristian M?rie? wrote: >>>> >>>> > Also there's the issue about not being able to implement a true proxy >>>> (as >>>> > outlined before). >>>> >>>> Proxies are a bit of a pain. But you can create your own callable >>>> function. >>>> >>>> Something like (untested): >>>> >>>> def callable(obj): >>>> try: >>>> func = obj.__call__ >>>> return True >>>> except AttributeError: >>>> return False >>>> >>>> -- >>>> ~Ethan~ >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 18 00:46:27 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Apr 2015 08:46:27 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: <5531869C.8090804@gmail.com> References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> Message-ID: On Sat, Apr 18, 2015 at 8:18 AM, Yury Selivanov wrote: > On 2015-04-17 6:00 PM, Chris Angelico wrote: >> >> On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov >> wrote: >>> running = True >>> while running: >>> try: >>> TARGET = await type(iter).__anext__(iter) >>> except StopAsyncIteration: >>> running = False >>> else: >>> BLOCK >>> else: >>> BLOCK2 >> >> (Not sure why you don't just use "break" instead of "running = False"? >> Maybe I'm blind to some distinction here.) > > Because if you use break, the "else: BLOCK2" won't execute, but > it should, as StopAsyncIteration is not a regular exception, but > something to signal that the iteration is over. Ah, gotcha. Carry on! >> Can't a coroutine simply use 'return'? I'm not hugely familiar with >> them, but I'm not understanding here why you need a completely >> separate exception - and, more importantly, why you wouldn't have the >> same consideration of "StopAsyncIteration leakage". It ought to be >> possible for the coroutine to return, StopIteration be synthesized, >> and the __anext__ function to bubble that out - exactly the way it >> does for generators today. > > > Unfortunately, there is no way we can enable StopIteration to do > the same thing that StopAsyncIteration does. > > 'return' in coroutines is implemented with 'StopIteration' exception > internally, so there is no way (without ugly hacks) to distinguish > before plain returns and desire to stop the iteration. All you need to do is have the same StopIteration -> RuntimeError handling that PEP 479 proposes, and then you can confidently accept a StopIteration coming out of __anext__ as a signal that the coroutine returned. It might require a small hack during the transitional time (eg "all functions defined with 'async def' behave as if defined in the presence of a 'from __future__ import generator_stop' directive"), but once PEP 479 handling becomes standard, generators and coroutines will all always follow this distinction. Generators either yield or return, coroutines either await or return, __[a]next__ methods either return or raise StopIteration, and any other issues become exceptions that bubble up. Since you're already utilizing a lot of generator-like behaviour, it shouldn't be a problem to handle exceptions the same way. The logic around a generator's __next__ method is broadly thus: * Resume the function's execution * If it returned: Synthesize a StopIteration. * Else if it raised StopIteration: Raise RuntimeError. * Else if it raised anything else: Let it bubble up. * Else it yielded something, so return it. There's no conflict here, because they're checked in strict order. It ought to be possible to do the same thing. The only reason for messy hacks is to support Python 2.x, where "return X" (for any X other than None) doesn't work; given that you're adding new syntax here, the concern shouldn't apply. > Let's see what python-ideas thinks about it. I'm fine if everybody > wants __future__ imports. I will only have to rollback my changes > in tokenizer.c and change few tokens in Grammar to make the > reference implementation work. In case it wasn't clear from my previous post, I'm +1 on using a __future__ import. Victor's idea of an optional directive seems interesting, but I'm not sure how useful it'd be in reality; does complicating the rules offer more benefit than simply having a keyword governed by __future__? >>> Comprehensions >>> -------------- >>> >>> For the sake of restricting the broadness of this PEP there is no new >>> syntax >>> for asynchronous comprehensions. This should be considered in a separate >>> PEP, >>> if there is a strong demand for this feature. >> >> Likewise asynchronous lambda functions (or maybe I just missed it). > > Right. Right now I'm just not sure they would be useful at all, > whereas I can imagine some usecases for comprehensions ;) Sure. I'm making no statement as to whether or not async lambda is worth implementing, but just that it's another subproposal that isn't being dealt with in this PEP. ChrisA From abarnert at yahoo.com Sat Apr 18 00:44:47 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 17 Apr 2015 15:44:47 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <55318855.7040905@gmail.com> References: <553157F2.8060408@gmail.com> <55318855.7040905@gmail.com> Message-ID: On Apr 17, 2015, at 15:25, Yury Selivanov wrote: > > >> On 2015-04-17 6:15 PM, Andrew Barnert wrote: >> Could you show the best yield from parallel for the example at the end, so it's easier to see the readability benefits? >> >> Or, even better, provide a simple example that uses async for or async with and the yield from parallel, so it's easier to see the (I'm assuming even bigger, by a long shot) readability benefits? > I don't have a real example, but we're getting there I don't mean something that big; just the simple example at the end of the PEP, possibly with an "async for" or "async with" added if there's a reasonable way to do so while still staying that trivial, shown the "new way" vs. the "old way". > > Andrew Svetlov made some experiments porting his aiopg library > to the new syntax. So perhaps early next week we will have > a real example of real code. > > You can also experiment yourself with the implementation. > > Thanks you, > > Yury > From rosuav at gmail.com Sat Apr 18 00:52:34 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Apr 2015 08:52:34 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150417222410.GA25999@k3> References: <553157F2.8060408@gmail.com> <20150417222410.GA25999@k3> Message-ID: On Sat, Apr 18, 2015 at 8:24 AM, David Wilson wrote: > Is there a need or desire for "async lambda ...:"? Notably, it's > possible though fairly useless to define a lambda generator: > "lambda x: (yield from x)" works today. Possible, indeed, but I suspect it's pretty rare. That's why bugs like this can lurk around - virtually nobody uses lambda to write generators: http://bugs.python.org/issue23192 And it wasn't production code that led me to run into that issue, it was solely from playing with the syntax. ChrisA From yselivanov.ml at gmail.com Sat Apr 18 00:56:37 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 18:56:37 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> Message-ID: <55318FA5.6010403@gmail.com> Chris, On 2015-04-17 6:46 PM, Chris Angelico wrote: > On Sat, Apr 18, 2015 at 8:18 AM, Yury Selivanov wrote: >> On 2015-04-17 6:00 PM, Chris Angelico wrote: >>> On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov >>> wrote: >>> [...] >>> Can't a coroutine simply use 'return'? I'm not hugely familiar with >>> them, but I'm not understanding here why you need a completely >>> separate exception - and, more importantly, why you wouldn't have the >>> same consideration of "StopAsyncIteration leakage". It ought to be >>> possible for the coroutine to return, StopIteration be synthesized, >>> and the __anext__ function to bubble that out - exactly the way it >>> does for generators today. >> >> Unfortunately, there is no way we can enable StopIteration to do >> the same thing that StopAsyncIteration does. >> >> 'return' in coroutines is implemented with 'StopIteration' exception >> internally, so there is no way (without ugly hacks) to distinguish >> before plain returns and desire to stop the iteration. > All you need to do is have the same StopIteration -> RuntimeError > handling that PEP 479 proposes, and then you can confidently accept a > StopIteration coming out of __anext__ as a signal that the coroutine > returned. It might require a small hack during the transitional time > (eg "all functions defined with 'async def' behave as if defined in > the presence of a 'from __future__ import generator_stop' directive"), > but once PEP 479 handling becomes standard, generators and coroutines > will all always follow this distinction. Generators either yield or > return, coroutines either await or return, __[a]next__ methods either > return or raise StopIteration, and any other issues become exceptions > that bubble up. > > Since you're already utilizing a lot of generator-like behaviour, it > shouldn't be a problem to handle exceptions the same way. The logic > around a generator's __next__ method is broadly thus: > > * Resume the function's execution > * If it returned: Synthesize a StopIteration. > * Else if it raised StopIteration: Raise RuntimeError. > * Else if it raised anything else: Let it bubble up. > * Else it yielded something, so return it. > > There's no conflict here, because they're checked in strict order. It > ought to be possible to do the same thing. The only reason for messy > hacks is to support Python 2.x, where "return X" (for any X other than > None) doesn't work; given that you're adding new syntax here, the > concern shouldn't apply. You forget that '__anext__' can be implemented as 'async def' function, as well as a regular 'def' returning an *awaitable* object, such as asyncio.Future or generator wrapped in 'types.async_def'. All in all, I think that it's probably possible to implement what you suggest. But I also believe that having a new built-in exception for the new protocol is a 100% better solution, simpler to implement and explain in documentation. > >> Let's see what python-ideas thinks about it. I'm fine if everybody >> wants __future__ imports. I will only have to rollback my changes >> in tokenizer.c and change few tokens in Grammar to make the >> reference implementation work. > In case it wasn't clear from my previous post, I'm +1 on using a > __future__ import. Victor's idea of an optional directive seems > interesting, but I'm not sure how useful it'd be in reality; does > complicating the rules offer more benefit than simply having a keyword > governed by __future__? I'm OK with __future__ import. And I'll be extremely happy if that's the only thing we'll be all discussing here ;-) Let's see how the discussion goes, and if everybody on the list wants __future__ and Guido approves, I'll update the PEP and ref implementation! > >>>> Comprehensions >>>> -------------- >>>> >>>> For the sake of restricting the broadness of this PEP there is no new >>>> syntax >>>> for asynchronous comprehensions. This should be considered in a separate >>>> PEP, >>>> if there is a strong demand for this feature. >>> Likewise asynchronous lambda functions (or maybe I just missed it). >> Right. Right now I'm just not sure they would be useful at all, >> whereas I can imagine some usecases for comprehensions ;) > Sure. I'm making no statement as to whether or not async lambda is > worth implementing, but just that it's another subproposal that isn't > being dealt with in this PEP. I think you're right, I need to add 'async lambda' to the 'Design Considerations' section. Thanks! Yury From yselivanov.ml at gmail.com Sat Apr 18 00:57:52 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 18:57:52 -0400 Subject: [Python-ideas] Fwd: Re: async/await in Python In-Reply-To: <4dd31123-ef09-4ee4-8773-4d318b1a3c88@googlegroups.com> References: <4dd31123-ef09-4ee4-8773-4d318b1a3c88@googlegroups.com> Message-ID: <55318FF0.20601@gmail.com> An email from Ludovic Gasc that he asked me to forward to python-ideas. It gets rejected or something, I'm not sure. -------- Forwarded Message -------- Subject: Re: [Python-ideas] async/await in Python Date: Fri, 17 Apr 2015 15:38:21 -0700 (PDT) From: Ludovic Gasc To: python-ideas at googlegroups.com CC: python-ideas at python.org, yselivanov.ml at gmail.com As a lambda end-user of AsyncIO, it seems promising. As I said to Andrew Svetlov during PyCON 2015, I really prefer this new syntax. For a new comer, it will be more easier to handle because you don't need to explain the first time what is a decorator, a yield, and a yield from in the same time that Futures, Coroutines and Tasks. It's a (small) distraction about Python internal plumber about AsyncIO implementation, instead to concentrate on async business logic implementation. Nevertheless, the only small fear I have is a potential "schism" in AsyncIO community like we have with Python 2/3. For end-products source code, no problems: For new projects you can use the new syntax, or migrate easily old projects, thanks to compatibility plan. But for open source libraries, you must keep old syntax is you want Python 3.3+ compatibility. I know it's a small change compare to Python 3 changes, and AsyncIO community is enough small for now and almost AsyncIO users are early adopters for now: It means it will be more fluid for the shift. Because async keyword integration will change the end-user source code, I vote for an integration in Python 3.5 (not 3.6): it will reduce the waiting time for the end-users to use that. In an ideal world, it should be useful if we could also use that with Python 3.3 and 3.4 to help migration, because mainstream Linux distributions for production (Debian Jessie, Ubuntu 14.04 and CentOS/RHEL 7) have packages for Python 3.4, not Python 3.5, and it will change only for the new versions of theses distributions. Nevertheless, I understand the technical and political reasons that it's impossible to do that: It's an internal change of CPython, can't be added via a third library. Hopefully, it's really easy to compile CPython, an AsyncIO early adopter can't be afraid to do that ;-) -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ On Friday, April 17, 2015 at 8:59:29 PM UTC+2, Yury Selivanov wrote: > > Hello python-ideas, > > Here's my proposal to add async/await in Python. > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > and I hope that this PEP can be another big step. Who knows, maybe it > will be one of the reasons to drive people's interest towards Python 3. > > > PEP: XXX > Title: Coroutines with async and await syntax > Version: $Revision$ > Last-Modified: $Date$ > Author: Yury Selivanov > > Discussions-To: Python-Dev > > Python-Version: 3.5 > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 09-Apr-2015 > Post-History: > Resolution: > > > Abstract > ======== > > This PEP introduces new syntax for coroutines, asynchronous ``with`` > statements and ``for`` loops. The main motivation behind this proposal > is to > streamline writing and maintaining asynchronous code, as well as to > simplify > previously hard to implement code patterns. > > > Rationale and Goals > =================== > > Current Python supports implementing coroutines via generators (PEP 342), > further enhanced by the ``yield from`` syntax introduced in PEP 380. > This approach has a number of shortcomings: > > * it is easy to confuse coroutines with regular generators, since they > share > the same syntax; async libraries often attempt to alleviate this by > using > decorators (e.g. ``@asyncio.coroutine`` [1]_); > > * it is not possible to natively define a coroutine which has no ``yield`` > or ``yield from`` statements, again requiring the use of decorators to > fix potential refactoring issues; > > * support for asynchronous calls is limited to expressions where > ``yield`` is > allowed syntactically, limiting the usefulness of syntactic features, > such > as ``with`` and ``for`` statements. > > This proposal makes coroutines a native Python language feature, and > clearly > separates them from generators. This removes generator/coroutine > ambiguity, > and makes it possible to reliably define coroutines without reliance on a > specific library. This also enables linters and IDEs to improve static > code > analysis and refactoring. > > Native coroutines and the associated new syntax features make it possible > to define context manager and iteration protocols in asynchronous terms. > As shown later in this proposal, the new ``async with`` statement lets > Python > programs perform asynchronous calls when entering and exiting a runtime > context, and the new ``async for`` statement makes it possible to perform > asynchronous calls in iterators. > > > Specification > ============= > > This proposal introduces new syntax and semantics to enhance coroutine > support > in Python, it does not change the internal implementation of coroutines, > which > are still based on generators. > > It is strongly suggested that the reader understands how coroutines are > implemented in Python (PEP 342 and PEP 380). It is also recommended to > read > PEP 3156 (asyncio framework). > > From this point in this document we use the word *coroutine* to refer to > functions declared using the new syntax. *generator-based coroutine* is > used > where necessary to refer to coroutines that are based on generator syntax. > > > New Coroutine Declaration Syntax > -------------------------------- > > The following new syntax is used to declare a coroutine:: > > async def read_data(db): > pass > > Key properties of coroutines: > > * Coroutines are always generators, even if they do not contain ``await`` > expressions. > > * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions > in > an ``async`` function. > > * Internally, a new code object flag - ``CO_ASYNC`` - is introduced to > enable > runtime detection of coroutines (and migrating existing code). > All coroutines have both ``CO_ASYNC`` and ``CO_GENERATOR`` flags set. > > * Regular generators, when called, return a *generator object*; similarly, > coroutines return a *coroutine object*. > > * ``StopIteration`` exceptions are not propagated out of coroutines, and > are > replaced with a ``RuntimeError``. For regular generators such behavior > requires a future import (see PEP 479). > > > types.async_def() > ----------------- > > A new function ``async_def(gen)`` is added to the ``types`` module. It > applies ``CO_ASYNC`` flag to the passed generator's code object, so that it > returns a *coroutine object* when called. > > This feature enables an easy upgrade path for existing libraries. > > > Await Expression > ---------------- > > The following new ``await`` expression is used to obtain a result of > coroutine > execution:: > > async def read_data(db): > data = await db.fetch('SELECT ...') > ... > > ``await``, similarly to ``yield from``, suspends execution of ``read_data`` > coroutine until ``db.fetch`` *awaitable* completes and returns the result > data. > > It uses the ``yield from`` implementation with an extra step of > validating its > argument. ``await`` only accepts an *awaitable*, which can be one of: > > * A *coroutine object* returned from a coroutine or a generator > decorated with > ``types.async_def()``. > > * An object with an ``__await__`` method returning an iterator. > > Any ``yield from`` chain of calls ends with a ``yield``. This is a > fundamental mechanism of how *Futures* are implemented. Since, > internally, > coroutines are a special kind of generators, every ``await`` is > suspended by > a ``yield`` somewhere down the chain of ``await`` calls (please refer > to PEP > 3156 for a detailed explanation.) > > To enable this behavior for coroutines, a new magic method called > ``__await__`` is added. In asyncio, for instance, to enable Future > objects > in ``await`` statements, the only change is to add ``__await__ = > __iter__`` > line to ``asyncio.Future`` class. > > Objects with ``__await__`` method are called *Future-like* objects in > the > rest of this PEP. > > Also, please note that ``__aiter__`` method (see its definition > below) cannot > be used for this purpose. It is a different protocol, and would be like > using ``__iter__`` instead of ``__call__`` for regular callables. > > It is a ``SyntaxError`` to use ``await`` outside of a coroutine. > > > Asynchronous Context Managers and "async with" > ---------------------------------------------- > > An *asynchronous context manager* is a context manager that is able to > suspend > execution in its *enter* and *exit* methods. > > To make this possible, a new protocol for asynchronous context managers is > proposed. Two new magic methods are added: ``__aenter__`` and > ``__aexit__``. > Both must return an *awaitable*. > > An example of an asynchronous context manager:: > > class AsyncContextManager: > async def __aenter__(self): > await log('entering context') > > async def __aexit__(self, exc_type, exc, tb): > await log('exiting context') > > > New Syntax > '''''''''' > > A new statement for asynchronous context managers is proposed:: > > async with EXPR as VAR: > BLOCK > > > which is semantically equivalent to:: > > mgr = (EXPR) > aexit = type(mgr).__aexit__ > aenter = type(mgr).__aenter__(mgr) > exc = True > > try: > try: > VAR = await aenter > BLOCK > except: > exc = False > exit_res = await aexit(mgr, *sys.exc_info()) > if not exit_res: > raise > > finally: > if exc: > await aexit(mgr, None, None, None) > > > As with regular ``with`` statements, it is possible to specify multiple > context > managers in a single ``async with`` statement. > > It is an error to pass a regular context manager without ``__aenter__`` and > ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use > ``async with`` outside of a coroutine. > > > Example > ''''''' > > With asynchronous context managers it is easy to implement proper database > transaction managers for coroutines:: > > async def commit(session, data): > ... > > async with session.transaction(): > ... > await session.update(data) > ... > > Code that needs locking also looks lighter:: > > async with lock: > ... > > instead of:: > > with (yield from lock): > ... > > > Asynchronous Iterators and "async for" > -------------------------------------- > > An *asynchronous iterable* is able to call asynchronous code in its *iter* > implementation, and *asynchronous iterator* can call asynchronous code > in its > *next* method. To support asynchronous iteration: > > 1. An object must implement an ``__aiter__`` method returning an > *awaitable* > resulting in an *asynchronous iterator object*. > > 2. An *asynchronous iterator object* must implement an ``__anext__`` method > returning an *awaitable*. > > 3. To stop iteration```__anext__`` must raise a ``StopAsyncIteration`` > exception. > > An example of asynchronous iterable:: > > class AsyncIterable: > async def __aiter__(self): > return self > > async def __anext__(self): > data = await self.fetch_data() > if data: > return data > else: > raise StopAsyncIteration > > async def fetch_data(self): > ... > > > New Syntax > '''''''''' > > A new statement for iterating through asynchronous iterators is proposed:: > > async for TARGET in ITER: > BLOCK > else: > BLOCK2 > > which is semantically equivalent to:: > > iter = (ITER) > iter = await type(iter).__aiter__(iter) > running = True > while running: > try: > TARGET = await type(iter).__anext__(iter) > except StopAsyncIteration: > running = False > else: > BLOCK > else: > BLOCK2 > > > It is an error to pass a regular iterable without ``__aiter__`` method to > ``async for``. It is a ``SyntaxError`` to use ``async for`` outside of a > coroutine. > > As for with regular ``for`` statement, ``async for`` has an optional > ``else`` > clause. > > > Example 1 > ''''''''' > > With asynchronous iteration protocol it is possible to asynchronously > buffer > data during iteration:: > > async for data in cursor: > ... > > Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows > of data from a database after every ``N`` iterations. > > The following code illustrates new asynchronous iteration protocol:: > > class Cursor: > def __init__(self): > self.buffer = collections.deque() > > def _prefetch(self): > ... > > async def __aiter__(self): > return self > > async def __anext__(self): > if not self.buffer: > self.buffer = await self._prefetch() > if not self.buffer: > raise StopAsyncIteration > return self.buffer.popleft() > > then the ``Cursor`` class can be used as follows:: > > async for row in Cursor(): > print(row) > > which would be equivalent to the following code:: > > i = await Cursor().__aiter__() > while True: > try: > row = await i.__anext__() > except StopAsyncIteration: > break > else: > print(row) > > > Example 2 > ''''''''' > > The following is a utility class that transforms a regular iterable to an > asynchronous one. While this is not a very useful thing to do, the code > illustrates the relationship between regular and asynchronous iterators. > > :: > > class AsyncIteratorWrapper: > def __init__(self, obj): > self._it = iter(obj) > > async def __aiter__(self): > return self > > async def __anext__(self): > try: > value = next(self._it) > except StopIteration: > raise StopAsyncIteration > return value > > data = "abc" > it = AsyncIteratorWrapper("abc") > async for item in it: > print(it) > > > Why StopAsyncIteration? > ''''''''''''''''''''''' > > Coroutines are still based on generators internally. So, before PEP > 479, there > was no fundamental difference between > > :: > > def g1(): > yield from fut > return 'spam' > > and > > :: > > def g2(): > yield from fut > raise StopIteration('spam') > > And since PEP 479 is accepted and enabled by default for coroutines, the > following example will have its ``StopIteration`` wrapped into a > ``RuntimeError`` > > :: > > async def a1(): > await fut > raise StopIteration('spam') > > The only way to tell the outside code that the iteration has ended is to > raise > something other than ``StopIteration``. Therefore, a new built-in > exception > class ``StopAsyncIteration`` was added. > > Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions > raised > in coroutines are wrapped in ``RuntimeError``. > > > Debugging Features > ------------------ > > One of the most frequent mistakes that people make when using generators as > coroutines is forgetting to use ``yield from``:: > > @asyncio.coroutine > def useful(): > asyncio.sleep(1) # this will do noting without 'yield from' > > For debugging this kind of mistakes there is a special debug mode in > asyncio, > in which ``@coroutine`` decorator wraps all functions with a special object > with a destructor logging a warning. Whenever a wrapped generator gets > garbage > collected, a detailed logging message is generated with information > about where > exactly the decorator function was defined, stack trace of where it was > collected, etc. Wrapper object also provides a convenient ``__repr__`` > function with detailed information about the generator. > > The only problem is how to enable these debug capabilities. Since debug > facilities should be a no-op in production mode, ``@coroutine`` > decorator makes > the decision of whether to wrap or not to wrap based on an OS environment > variable ``PYTHONASYNCIODEBUG``. This way it is possible to run asyncio > programs with asyncio's own functions instrumented. > ``EventLoop.set_debug``, a > different debug facility, has no impact on ``@coroutine`` decorator's > behavior. > > With this proposal, coroutines is a native, distinct from generators, > concept. A new method ``set_async_wrapper`` is added to the ``sys`` > module, > with which frameworks can provide advanced debugging facilities. > > It is also important to make coroutines as fast and efficient as possible, > therefore there are no debug features enabled by default. > > Example:: > > async def debug_me(): > await asyncio.sleep(1) > > def async_debug_wrap(generator): > return asyncio.AsyncDebugWrapper(generator) > > sys.set_async_wrapper(async_debug_wrap) > > debug_me() # <- this line will likely GC the coroutine object and > # trigger AsyncDebugWrapper's code. > > assert isinstance(debug_me(), AsyncDebugWrapper) > > sys.set_async_wrapper(None) # <- this unsets any previously set > wrapper > assert not isinstance(debug_me(), AsyncDebugWrapper) > > If ``sys.set_async_wrapper()`` is called twice, the new wrapper replaces > the > previous wrapper. ``sys.set_async_wrapper(None)`` unsets the wrapper. > > > Glossary > ======== > > :Coroutine: > A coroutine function, or just "coroutine", is declared with ``async > def``. > It uses ``await`` and ``return value``; see `New Coroutine Declaration > Syntax`_ for details. > > :Coroutine object: > Returned from a coroutine function. See `Await Expression`_ for > details. > > :Future-like object: > An object with an ``__await__`` method. It is consumed by > ``await`` in a > coroutine. A coroutine waiting for a Future-like object is > suspended until > the Future-like object's ``__await__`` completes. ``await`` > returns the > result of the Future-like object. See `Await Expression`_ for > details. > > :Awaitable: > A *future-like* object or a *coroutine object*. See `Await > Expression`_ > for details. > > :Generator-based coroutine: > Coroutines based in generator syntax. Most common example is > ``@asyncio.coroutine``. > > :Asynchronous context manager: > An asynchronous context manager has ``__aenter__`` and ``__aexit__`` > methods > and can be used with ``async with``. See > `Asynchronous Context Managers and "async with"`_ for details. > > :Asynchronous iterable: > An object with an ``__aiter__`` method, which must return an > *asynchronous > iterator* object. Can be used with ``async for``. See > `Asynchronous Iterators and "async for"`_ for details. > > :Asynchronous iterator: > An asynchronous iterator has an ``__anext__`` method.See > `Asynchronous Iterators and "async for"`_ for details. > > > List of functions and methods > ============================= > > ================= ======================================= ================= > Method Can contain Can't contain > ================= ======================================= ================= > async def func await, return value yield, yield > from > async def __a*__ await, return value yield, yield > from > def __a*__ return Future-like await > def __await__ yield, yield from, return iterable await > generator yield, yield from, return value await > ================= ======================================= ================= > > Where: > > * ""async def func": coroutine; > > * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, > ``__aexit__`` defined with the ``async`` keyword; > > * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` > defined without the ``async`` keyword, must return an *awaitable*; > > * "def __await__": ``__await__`` method to implement *Future-like* objects; > > * generator: a "regular" generator, function defined with ``def`` and which > contains a least one ``yield`` or ``yield from`` expression. > > *Future-like* is an object with an ``__await__`` method, see > `Await Expression`_ section for details. > > > Transition Plan > =============== > > To avoid backwards compatibility issues with ``async`` and ``await`` > keywords, > it was decided to modify ``tokenizer.c`` in such a way, that it: > > * recognizes ``async def`` name tokens combination (start of a coroutine); > > * keeps track of regular functions and coroutines; > > * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with > ``AWAIT`` when in the process of yielding tokens for coroutines. > > This approach allows for seamless combination of new syntax features (all > of > them available only in ``async`` functions) with any existing code. > > An example of having "async def" and "async" attribute in one piece of > code:: > > class Spam: > async = 42 > > async def ham(): > print(getattr(Spam, 'async')) > > # The coroutine can be executed and will print '42' > > > Backwards Compatibility > ----------------------- > > The only backwards incompatible change is an extra argument ``is_async`` to > ``FunctionDef`` AST node. But since it is a documented fact that the > structure > of AST nodes is an implementation detail and subject to change, this > should not > be considered a serious issue. > > > Grammar Updates > --------------- > > Grammar changes are also fairly minimal:: > > await_expr: AWAIT test > await_stmt: await_expr > > decorated: decorators (classdef | funcdef | async_funcdef) > async_funcdef: ASYNC funcdef > > async_stmt: ASYNC (funcdef | with_stmt) # will add for_stmt later > > compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt > | funcdef | classdef | decorated | async_stmt) > > atom: ('(' [yield_expr|await_expr|testlist_comp] ')' | > '[' [testlist_comp] ']' | > '{' [dictorsetmaker] '}' | > NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False?) > > expr_stmt: testlist_star_expr (augassign > (yield_expr|await_expr|testlist) | > ('=' (yield_expr|await_expr|testlist_star_expr))*) > > > Transition Period Shortcomings > ------------------------------ > > There is just one. > > Until ``async`` and ``await`` are not proper keywords, it is not > possible (or > at least very hard) to fix ``tokenizer.c`` to recognize them on the **same > line** with ``def`` keyword:: > > # async and await will always be parsed as variables > > async def outer(): # 1 > def nested(a=(await fut)): > pass > > async def foo(): return (await fut) # 2 > > Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, > a > ``SyntaxError`` will be raised. > > To workaround these issues, the above examples can be easily rewritten to a > more readable form:: > > async def outer(): # 1 > a_default = await fut > def nested(a=a_default): > pass > > async def foo(): # 2 > return (await fut) > > This limitation will go away as soon as ``async`` and ``await`` ate proper > keywords. Or if it's decided to use a future import for this PEP. > > > Deprecation Plans > ----------------- > > ``async`` and ``await`` names will be softly deprecated in CPython 3.5 > and 3.6. > In 3.7 we will transform them to proper keywords. Making ``async`` and > ``await`` proper keywords before 3.7 might make it harder for people to > port > their code to Python 3. > > > asyncio > ------- > > ``asyncio`` module was adapted and tested to work with coroutines and new > statements. Backwards compatibility is 100% preserved. > > The required changes are mainly: > > 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.async_def()`` > function. > > 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. > > 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate > ``async()`` function. > > > Design Considerations > ===================== > > No implicit wrapping in Futures > ------------------------------- > > There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key > difference is that JavaScript "async functions" always return a Promise. > While > this approach has some advantages, it also implies that a new Promise > object is > created on each "async function" invocation. > > We could implement a similar functionality in Python, by wrapping all > coroutines in a Future object, but this has the following disadvantages: > > 1. Performance. A new Future object would be instantiated on each > coroutine > call. Moreover, this makes implementation of ``await`` expressions > slower > (disabling optimizations of ``yield from``). > > 2. A new built-in ``Future`` object would need to be added. > > 3. Coming up with a generic ``Future`` interface that is usable for any use > case in any framework is a very hard to solve problem. > > 4. It is not a feature that is used frequently, when most of the code is > coroutines. > > > Why "async" and "await" keywords > -------------------------------- > > async/await is not a new concept in programming languages: > > * C# has it since long time ago [5]_; > > * proposal to add async/await in ECMAScript 7 [2]_; > see also Traceur project [9]_; > > * Facebook's Hack/HHVM [6]_; > > * Google's Dart language [7]_; > > * Scala [8]_; > > * proposal to add async/await to C++ [10]_; > > * and many other less popular languages. > > This is a huge benefit, as some users already have experience with > async/await, > and because it makes working with many languages in one project easier > (Python > with ECMAScript 7 for instance). > > > Why "__aiter__" is a coroutine > ------------------------------ > > In principle, ``__aiter__`` could be a regular function. There are several > good reasons to make it a coroutine: > > * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` > methods are > coroutines, users would often make a mistake defining it as ``async`` > anyways; > > * there might be a need to run some asynchronous operations in > ``__aiter__``, > for instance to prepare DB queries or do some file operation. > > > Importance of "async" keyword > ----------------------------- > > While it is possible to just implement ``await`` expression and treat all > functions with at least one ``await`` as coroutines, this approach makes > APIs design, code refactoring and its long time support harder. > > Let's pretend that Python only has ``await`` keyword:: > > def useful(): > ... > await log(...) > ... > > def important(): > await useful() > > If ``useful()`` function is refactored and someone removes all ``await`` > expressions from it, it would become a regular python function, and all > code > that depends on it, including ``important()`` would be broken. To > mitigate this > issue a decorator similar to ``@asyncio.coroutine`` has to be introduced. > > > Why "async def" > --------------- > > For some people bare ``async name(): pass`` syntax might look more > appealing > than ``async def name(): pass``. It is certainly easier to type. But on > the > other hand, it breaks the symmetry between ``async def``, ``async with`` > and > ``async for``, where ``async`` is a modifier, stating that the statement is > asynchronous. It is also more consistent with the existing grammar. > > > Why not a __future__ import > --------------------------- > > ``__future__`` imports are inconvenient and easy to forget to add. Also, > they > are enabled for the whole source file. Consider that there is a big > project > with a popular module named "async.py". With future imports it is > required to > either import it using ``__import__()`` or ``importlib.import_module()`` > calls, > or to rename the module. The proposed approach makes it possible to > continue > using old code and modules without a hassle, while coming up with a > migration > plan for future python versions. > > > Why magic methods start with "a" > -------------------------------- > > New asynchronous magic methods ``__aiter__``, ``__anext__``, > ``__aenter__``, > and ``__aexit__`` all start with the same prefix "a". An alternative > proposal > is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. > However, to align new magic methods with the existing ones, such as > ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > > > Why not reuse existing magic names > ---------------------------------- > > An alternative idea about new asynchronous iterators and context > managers was > to reuse existing magic methods, by adding an ``async`` keyword to their > declarations:: > > class CM: > async def __enter__(self): # instead of __aenter__ > ... > > This approach has the following downsides: > > * it would not be possible to create an object that works in both > ``with`` and > ``async with`` statements; > > * it would look confusing and would require some implicit magic behind the > scenes in the interpreter; > > * one of the main points of this proposal is to make coroutines as simple > and foolproof as possible. > > > Comprehensions > -------------- > > For the sake of restricting the broadness of this PEP there is no new > syntax > for asynchronous comprehensions. This should be considered in a > separate PEP, > if there is a strong demand for this feature. > > > Performance > =========== > > Overall Impact > -------------- > > This proposal introduces no observable performance impact. Here is an > output > of python's official set of benchmarks [4]_: > > :: > > python perf.py -r -b default ../cpython/python.exe > ../cpython-aw/python.exe > > [skipped] > > Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: > Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 > x86_64 i386 > > Total CPU cores: 8 > > ### etree_iterparse ### > Min: 0.365359 -> 0.349168: 1.05x faster > Avg: 0.396924 -> 0.379735: 1.05x faster > Significant (t=9.71) > Stddev: 0.01225 -> 0.01277: 1.0423x larger > > The following not significant results are hidden, use -v to show them: > django_v2, 2to3, etree_generate, etree_parse, etree_process, > fastpickle, > fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. > > > Tokenizer modifications > ----------------------- > > There is no observable slowdown of parsing python files with the modified > tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated > 1000 > times) takes the same amount of time. > > > async/await > ----------- > > The following micro-benchmark was used to determine performance difference > between "async" functions and generators:: > > import sys > import time > > def binary(n): > if n <= 0: > return 1 > l = yield from binary(n - 1) > r = yield from binary(n - 1) > return l + 1 + r > > async def abinary(n): > if n <= 0: > return 1 > l = await abinary(n - 1) > r = await abinary(n - 1) > return l + 1 + r > > def timeit(gen, depth, repeat): > t0 = time.time() > for _ in range(repeat): > list(gen(depth)) > t1 = time.time() > print('{}({}) * {}: total {:.3f}s'.format( > gen.__name__, depth, repeat, t1-t0)) > > The result is that there is no observable performance difference. Minimum > timing of 3 runs > > :: > > abinary(19) * 30: total 12.985s > binary(19) * 30: total 12.953s > > Note that depth of 19 means 1,048,575 calls. > > > Reference Implementation > ======================== > > The reference implementation can be found here: [3]_. > > List of high-level changes and new protocols > -------------------------------------------- > > 1. New syntax for defining coroutines: ``async def`` and new ``await`` > keyword. > > 2. New ``__await__`` method for Future-like objects. > > 3. New syntax for asynchronous context managers: ``async with``. And > associated protocol with ``__aenter__`` and ``__aexit__`` methods. > > 4. New syntax for asynchronous iteration: ``async for``. And associated > protocol with ``__aiter__``, ``__aexit__`` and new built-in exception > ``StopAsyncIteration``. > > 5. New AST nodes: ``AsyncFor``, ``AsyncWith``, ``Await``; > ``FunctionDef`` AST > node got a new argument ``is_async``. > > 6. New functions: ``sys.set_async_wrapper(callback)`` and > ``types.async_def(gen)``. > > 7. New ``CO_ASYNC`` bit flag for code objects. > > While the list of changes and new things is not short, it is important to > understand, that most users will not use these features directly. It is > intended to be used in frameworks and libraries to provide users with > convenient to use and unambiguous APIs with ``async def``, ``await``, > ``async > for`` and ``async with`` syntax. > > > Working example > --------------- > > All concepts proposed in this PEP are implemented [3]_ and can be tested. > > :: > > import asyncio > > > async def echo_server(): > print('Serving on localhost:8000') > await asyncio.start_server(handle_connection, 'localhost', 8000) > > > async def handle_connection(reader, writer): > print('New connection...') > > while True: > data = await reader.read(8192) > > if not data: > break > > print('Sending {:.10}... back'.format(repr(data))) > writer.write(data) > > > loop = asyncio.get_event_loop() > loop.run_until_complete(echo_server()) > try: > loop.run_forever() > finally: > loop.close() > > > References > ========== > > .. [1] > https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine > > .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions > > .. [3] https://github.com/1st1/cpython/tree/await > > .. [4] https://hg.python.org/benchmarks > > .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx > > .. [6] http://docs.hhvm.com/manual/en/hack.async.php > > .. [7] https://www.dartlang.org/articles/await-async/ > > .. [8] http://docs.scala-lang.org/sips/pending/async.html > > .. [9] > > https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental > > .. [10] > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) > > > Acknowledgments > =============== > > I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew > Svetlov, > and ?ukasz Langa for their initial feedback. > > > Copyright > ========= > > This document has been placed in the public domain. > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > From rosuav at gmail.com Sat Apr 18 01:08:55 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Apr 2015 09:08:55 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: <55318FA5.6010403@gmail.com> References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> <55318FA5.6010403@gmail.com> Message-ID: On Sat, Apr 18, 2015 at 8:56 AM, Yury Selivanov wrote: > You forget that '__anext__' can be implemented as 'async def' > function, as well as a regular 'def' returning an *awaitable* > object, such as asyncio.Future or generator wrapped in > 'types.async_def'. Oh. Hrm. Good point. There is still the concern that StopAsyncIteration bubbling will be just as much a problem as StopIteration bubbling is, but you're in a better position than I am to look into that. ChrisA From yselivanov.ml at gmail.com Sat Apr 18 01:26:43 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 17 Apr 2015 19:26:43 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> <55318FA5.6010403@gmail.com> Message-ID: <553196B3.4080101@gmail.com> On 2015-04-17 7:08 PM, Chris Angelico wrote: > On Sat, Apr 18, 2015 at 8:56 AM, Yury Selivanov wrote: >> You forget that '__anext__' can be implemented as 'async def' >> function, as well as a regular 'def' returning an *awaitable* >> object, such as asyncio.Future or generator wrapped in >> 'types.async_def'. > Oh. Hrm. Good point. > > There is still the concern that StopAsyncIteration bubbling will be > just as much a problem as StopIteration bubbling is, but you're in a > better position than I am to look into that. I couldn't come up with an example where StopAsyncIteration would have same "bubbling" problems as StopIteration (I tried to). If someone can come up with a good case that would definitely trigger more thinking about that. FWIW, there is a PEP implementation, so it's possible to explore and prototype. Best, Yury From tjreedy at udel.edu Sat Apr 18 03:04:56 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 17 Apr 2015 21:04:56 -0400 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: On 4/17/2015 5:19 PM, Ionel Cristian M?rie? wrote: > __add__ as a property/descriptor seems to work fine, eg: > > >>> class C: > ... @property > ... def __add__(self): > ... return lambda other: [self, other] > ... > >>> C() + C() > [<__main__.C object at 0x0000000003652AC8>, <__main__.C object at > 0x0000000003652CC0>] > > > Am I missing something? An instance method is neither a class method nor a static method. It is also not a property. In your example above, __add__ is a property and not an instance method. This violate the Reference Manual definition of the reserved name '__add__' as an instance method. If f is an one-parameter instance method of class C and c is an instance of C, c.f() == C.f(c). This is not true for a property f, as C.f is not callable, and c.f is only callable if the value of the property is callable. -- What you have done above is to curry what is intended to be a binary instance method into a pair of functions, with the outer, function-returning function getting the original name. Having this curried pair implement the binary operator is quite clever. It works because CPython apparently implements c1 + c2, at least for user class instances, as c1.__add__(c2), instead of the presumed equivalent C.__add__(c1, c2). c1.__add__ is normally a bound method with c1 and C.__add__ as attributes. c1.__add__(c2) then calls C.__add__(c1, c2). In your example, c1.__add__ is a unary function with a captured nonlocal object. Calling it with a second object combines (adds) the two objects. The possibility of @property making this work is new to me. -- Terry Jan Reedy From oreilldf at gmail.com Sat Apr 18 03:31:20 2015 From: oreilldf at gmail.com (Dan O'Reilly) Date: Fri, 17 Apr 2015 21:31:20 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: This is exciting! I expect this will eliminate some of the confusion I see from new users of asyncio on StackOverflow. I'm curious about what this means for asyncio best practices from 3.5 onward, though. Is the use of @asyncio.coroutine/yield from being deprecated in asyncio once this feature is added? I guess anyone that needs to write backwards compatible code would need to continue using the old way, so perhaps not yet? At the very least the docs should probably mention that the async/wait syntax is available/preferred for code that only needs to support 3.5+. If you're going the deprecation route, the docs should be updated to use async/await, and probably just include a note stating that the old @asyncio.coroutine/yield from method is required to support 3.4 and lower. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Sat Apr 18 04:09:18 2015 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 17 Apr 2015 19:09:18 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: <3455689F-F60F-48BA-ABBE-189CAE1F1E10@langa.pl> Obviously +1. > On Apr 17, 2015, at 11:58 AM, Yury Selivanov wrote: > > Hello python-ideas, > > Here's my proposal to add async/await in Python. > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > and I hope that this PEP can be another big step. Who knows, maybe it > will be one of the reasons to drive people's interest towards Python 3. > > > PEP: XXX > Title: Coroutines with async and await syntax > Version: $Revision$ > Last-Modified: $Date$ > Author: Yury Selivanov > Discussions-To: Python-Dev > Python-Version: 3.5 > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 09-Apr-2015 > Post-History: > Resolution: > > > Abstract > ======== > > This PEP introduces new syntax for coroutines, asynchronous ``with`` > statements and ``for`` loops. The main motivation behind this proposal is to > streamline writing and maintaining asynchronous code, as well as to simplify > previously hard to implement code patterns. > > > Rationale and Goals > =================== > > Current Python supports implementing coroutines via generators (PEP 342), > further enhanced by the ``yield from`` syntax introduced in PEP 380. > This approach has a number of shortcomings: > > * it is easy to confuse coroutines with regular generators, since they share > the same syntax; async libraries often attempt to alleviate this by using > decorators (e.g. ``@asyncio.coroutine`` [1]_); > > * it is not possible to natively define a coroutine which has no ``yield`` > or ``yield from`` statements, again requiring the use of decorators to > fix potential refactoring issues; > > * support for asynchronous calls is limited to expressions where ``yield`` is > allowed syntactically, limiting the usefulness of syntactic features, such > as ``with`` and ``for`` statements. > > This proposal makes coroutines a native Python language feature, and clearly > separates them from generators. This removes generator/coroutine ambiguity, > and makes it possible to reliably define coroutines without reliance on a > specific library. This also enables linters and IDEs to improve static code > analysis and refactoring. > > Native coroutines and the associated new syntax features make it possible > to define context manager and iteration protocols in asynchronous terms. > As shown later in this proposal, the new ``async with`` statement lets Python > programs perform asynchronous calls when entering and exiting a runtime > context, and the new ``async for`` statement makes it possible to perform > asynchronous calls in iterators. > > > Specification > ============= > > This proposal introduces new syntax and semantics to enhance coroutine support > in Python, it does not change the internal implementation of coroutines, which > are still based on generators. > > It is strongly suggested that the reader understands how coroutines are > implemented in Python (PEP 342 and PEP 380). It is also recommended to read > PEP 3156 (asyncio framework). > > From this point in this document we use the word *coroutine* to refer to > functions declared using the new syntax. *generator-based coroutine* is used > where necessary to refer to coroutines that are based on generator syntax. > > > New Coroutine Declaration Syntax > -------------------------------- > > The following new syntax is used to declare a coroutine:: > > async def read_data(db): > pass > > Key properties of coroutines: > > * Coroutines are always generators, even if they do not contain ``await`` > expressions. > > * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in > an ``async`` function. > > * Internally, a new code object flag - ``CO_ASYNC`` - is introduced to enable > runtime detection of coroutines (and migrating existing code). > All coroutines have both ``CO_ASYNC`` and ``CO_GENERATOR`` flags set. > > * Regular generators, when called, return a *generator object*; similarly, > coroutines return a *coroutine object*. > > * ``StopIteration`` exceptions are not propagated out of coroutines, and are > replaced with a ``RuntimeError``. For regular generators such behavior > requires a future import (see PEP 479). > > > types.async_def() > ----------------- > > A new function ``async_def(gen)`` is added to the ``types`` module. It > applies ``CO_ASYNC`` flag to the passed generator's code object, so that it > returns a *coroutine object* when called. > > This feature enables an easy upgrade path for existing libraries. > > > Await Expression > ---------------- > > The following new ``await`` expression is used to obtain a result of coroutine > execution:: > > async def read_data(db): > data = await db.fetch('SELECT ...') > ... > > ``await``, similarly to ``yield from``, suspends execution of ``read_data`` > coroutine until ``db.fetch`` *awaitable* completes and returns the result > data. > > It uses the ``yield from`` implementation with an extra step of validating its > argument. ``await`` only accepts an *awaitable*, which can be one of: > > * A *coroutine object* returned from a coroutine or a generator decorated with > ``types.async_def()``. > > * An object with an ``__await__`` method returning an iterator. > > Any ``yield from`` chain of calls ends with a ``yield``. This is a > fundamental mechanism of how *Futures* are implemented. Since, internally, > coroutines are a special kind of generators, every ``await`` is suspended by > a ``yield`` somewhere down the chain of ``await`` calls (please refer to PEP > 3156 for a detailed explanation.) > > To enable this behavior for coroutines, a new magic method called > ``__await__`` is added. In asyncio, for instance, to enable Future objects > in ``await`` statements, the only change is to add ``__await__ = __iter__`` > line to ``asyncio.Future`` class. > > Objects with ``__await__`` method are called *Future-like* objects in the > rest of this PEP. > > Also, please note that ``__aiter__`` method (see its definition below) cannot > be used for this purpose. It is a different protocol, and would be like > using ``__iter__`` instead of ``__call__`` for regular callables. > > It is a ``SyntaxError`` to use ``await`` outside of a coroutine. > > > Asynchronous Context Managers and "async with" > ---------------------------------------------- > > An *asynchronous context manager* is a context manager that is able to suspend > execution in its *enter* and *exit* methods. > > To make this possible, a new protocol for asynchronous context managers is > proposed. Two new magic methods are added: ``__aenter__`` and ``__aexit__``. > Both must return an *awaitable*. > > An example of an asynchronous context manager:: > > class AsyncContextManager: > async def __aenter__(self): > await log('entering context') > > async def __aexit__(self, exc_type, exc, tb): > await log('exiting context') > > > New Syntax > '''''''''' > > A new statement for asynchronous context managers is proposed:: > > async with EXPR as VAR: > BLOCK > > > which is semantically equivalent to:: > > mgr = (EXPR) > aexit = type(mgr).__aexit__ > aenter = type(mgr).__aenter__(mgr) > exc = True > > try: > try: > VAR = await aenter > BLOCK > except: > exc = False > exit_res = await aexit(mgr, *sys.exc_info()) > if not exit_res: > raise > > finally: > if exc: > await aexit(mgr, None, None, None) > > > As with regular ``with`` statements, it is possible to specify multiple context > managers in a single ``async with`` statement. > > It is an error to pass a regular context manager without ``__aenter__`` and > ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use > ``async with`` outside of a coroutine. > > > Example > ''''''' > > With asynchronous context managers it is easy to implement proper database > transaction managers for coroutines:: > > async def commit(session, data): > ... > > async with session.transaction(): > ... > await session.update(data) > ... > > Code that needs locking also looks lighter:: > > async with lock: > ... > > instead of:: > > with (yield from lock): > ... > > > Asynchronous Iterators and "async for" > -------------------------------------- > > An *asynchronous iterable* is able to call asynchronous code in its *iter* > implementation, and *asynchronous iterator* can call asynchronous code in its > *next* method. To support asynchronous iteration: > > 1. An object must implement an ``__aiter__`` method returning an *awaitable* > resulting in an *asynchronous iterator object*. > > 2. An *asynchronous iterator object* must implement an ``__anext__`` method > returning an *awaitable*. > > 3. To stop iteration```__anext__`` must raise a ``StopAsyncIteration`` > exception. > > An example of asynchronous iterable:: > > class AsyncIterable: > async def __aiter__(self): > return self > > async def __anext__(self): > data = await self.fetch_data() > if data: > return data > else: > raise StopAsyncIteration > > async def fetch_data(self): > ... > > > New Syntax > '''''''''' > > A new statement for iterating through asynchronous iterators is proposed:: > > async for TARGET in ITER: > BLOCK > else: > BLOCK2 > > which is semantically equivalent to:: > > iter = (ITER) > iter = await type(iter).__aiter__(iter) > running = True > while running: > try: > TARGET = await type(iter).__anext__(iter) > except StopAsyncIteration: > running = False > else: > BLOCK > else: > BLOCK2 > > > It is an error to pass a regular iterable without ``__aiter__`` method to > ``async for``. It is a ``SyntaxError`` to use ``async for`` outside of a > coroutine. > > As for with regular ``for`` statement, ``async for`` has an optional ``else`` > clause. > > > Example 1 > ''''''''' > > With asynchronous iteration protocol it is possible to asynchronously buffer > data during iteration:: > > async for data in cursor: > ... > > Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows > of data from a database after every ``N`` iterations. > > The following code illustrates new asynchronous iteration protocol:: > > class Cursor: > def __init__(self): > self.buffer = collections.deque() > > def _prefetch(self): > ... > > async def __aiter__(self): > return self > > async def __anext__(self): > if not self.buffer: > self.buffer = await self._prefetch() > if not self.buffer: > raise StopAsyncIteration > return self.buffer.popleft() > > then the ``Cursor`` class can be used as follows:: > > async for row in Cursor(): > print(row) > > which would be equivalent to the following code:: > > i = await Cursor().__aiter__() > while True: > try: > row = await i.__anext__() > except StopAsyncIteration: > break > else: > print(row) > > > Example 2 > ''''''''' > > The following is a utility class that transforms a regular iterable to an > asynchronous one. While this is not a very useful thing to do, the code > illustrates the relationship between regular and asynchronous iterators. > > :: > > class AsyncIteratorWrapper: > def __init__(self, obj): > self._it = iter(obj) > > async def __aiter__(self): > return self > > async def __anext__(self): > try: > value = next(self._it) > except StopIteration: > raise StopAsyncIteration > return value > > data = "abc" > it = AsyncIteratorWrapper("abc") > async for item in it: > print(it) > > > Why StopAsyncIteration? > ''''''''''''''''''''''' > > Coroutines are still based on generators internally. So, before PEP 479, there > was no fundamental difference between > > :: > > def g1(): > yield from fut > return 'spam' > > and > > :: > > def g2(): > yield from fut > raise StopIteration('spam') > > And since PEP 479 is accepted and enabled by default for coroutines, the > following example will have its ``StopIteration`` wrapped into a > ``RuntimeError`` > > :: > > async def a1(): > await fut > raise StopIteration('spam') > > The only way to tell the outside code that the iteration has ended is to raise > something other than ``StopIteration``. Therefore, a new built-in exception > class ``StopAsyncIteration`` was added. > > Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions raised > in coroutines are wrapped in ``RuntimeError``. > > > Debugging Features > ------------------ > > One of the most frequent mistakes that people make when using generators as > coroutines is forgetting to use ``yield from``:: > > @asyncio.coroutine > def useful(): > asyncio.sleep(1) # this will do noting without 'yield from' > > For debugging this kind of mistakes there is a special debug mode in asyncio, > in which ``@coroutine`` decorator wraps all functions with a special object > with a destructor logging a warning. Whenever a wrapped generator gets garbage > collected, a detailed logging message is generated with information about where > exactly the decorator function was defined, stack trace of where it was > collected, etc. Wrapper object also provides a convenient ``__repr__`` > function with detailed information about the generator. > > The only problem is how to enable these debug capabilities. Since debug > facilities should be a no-op in production mode, ``@coroutine`` decorator makes > the decision of whether to wrap or not to wrap based on an OS environment > variable ``PYTHONASYNCIODEBUG``. This way it is possible to run asyncio > programs with asyncio's own functions instrumented. ``EventLoop.set_debug``, a > different debug facility, has no impact on ``@coroutine`` decorator's behavior. > > With this proposal, coroutines is a native, distinct from generators, > concept. A new method ``set_async_wrapper`` is added to the ``sys`` module, > with which frameworks can provide advanced debugging facilities. > > It is also important to make coroutines as fast and efficient as possible, > therefore there are no debug features enabled by default. > > Example:: > > async def debug_me(): > await asyncio.sleep(1) > > def async_debug_wrap(generator): > return asyncio.AsyncDebugWrapper(generator) > > sys.set_async_wrapper(async_debug_wrap) > > debug_me() # <- this line will likely GC the coroutine object and > # trigger AsyncDebugWrapper's code. > > assert isinstance(debug_me(), AsyncDebugWrapper) > > sys.set_async_wrapper(None) # <- this unsets any previously set wrapper > assert not isinstance(debug_me(), AsyncDebugWrapper) > > If ``sys.set_async_wrapper()`` is called twice, the new wrapper replaces the > previous wrapper. ``sys.set_async_wrapper(None)`` unsets the wrapper. > > > Glossary > ======== > > :Coroutine: > A coroutine function, or just "coroutine", is declared with ``async def``. > It uses ``await`` and ``return value``; see `New Coroutine Declaration > Syntax`_ for details. > > :Coroutine object: > Returned from a coroutine function. See `Await Expression`_ for details. > > :Future-like object: > An object with an ``__await__`` method. It is consumed by ``await`` in a > coroutine. A coroutine waiting for a Future-like object is suspended until > the Future-like object's ``__await__`` completes. ``await`` returns the > result of the Future-like object. See `Await Expression`_ for details. > > :Awaitable: > A *future-like* object or a *coroutine object*. See `Await Expression`_ > for details. > > :Generator-based coroutine: > Coroutines based in generator syntax. Most common example is > ``@asyncio.coroutine``. > > :Asynchronous context manager: > An asynchronous context manager has ``__aenter__`` and ``__aexit__`` methods > and can be used with ``async with``. See > `Asynchronous Context Managers and "async with"`_ for details. > > :Asynchronous iterable: > An object with an ``__aiter__`` method, which must return an *asynchronous > iterator* object. Can be used with ``async for``. See > `Asynchronous Iterators and "async for"`_ for details. > > :Asynchronous iterator: > An asynchronous iterator has an ``__anext__`` method.See > `Asynchronous Iterators and "async for"`_ for details. > > > List of functions and methods > ============================= > > ================= ======================================= ================= > Method Can contain Can't contain > ================= ======================================= ================= > async def func await, return value yield, yield from > async def __a*__ await, return value yield, yield from > def __a*__ return Future-like await > def __await__ yield, yield from, return iterable await > generator yield, yield from, return value await > ================= ======================================= ================= > > Where: > > * ""async def func": coroutine; > > * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, > ``__aexit__`` defined with the ``async`` keyword; > > * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` > defined without the ``async`` keyword, must return an *awaitable*; > > * "def __await__": ``__await__`` method to implement *Future-like* objects; > > * generator: a "regular" generator, function defined with ``def`` and which > contains a least one ``yield`` or ``yield from`` expression. > > *Future-like* is an object with an ``__await__`` method, see > `Await Expression`_ section for details. > > > Transition Plan > =============== > > To avoid backwards compatibility issues with ``async`` and ``await`` keywords, > it was decided to modify ``tokenizer.c`` in such a way, that it: > > * recognizes ``async def`` name tokens combination (start of a coroutine); > > * keeps track of regular functions and coroutines; > > * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with > ``AWAIT`` when in the process of yielding tokens for coroutines. > > This approach allows for seamless combination of new syntax features (all of > them available only in ``async`` functions) with any existing code. > > An example of having "async def" and "async" attribute in one piece of code:: > > class Spam: > async = 42 > > async def ham(): > print(getattr(Spam, 'async')) > > # The coroutine can be executed and will print '42' > > > Backwards Compatibility > ----------------------- > > The only backwards incompatible change is an extra argument ``is_async`` to > ``FunctionDef`` AST node. But since it is a documented fact that the structure > of AST nodes is an implementation detail and subject to change, this should not > be considered a serious issue. > > > Grammar Updates > --------------- > > Grammar changes are also fairly minimal:: > > await_expr: AWAIT test > await_stmt: await_expr > > decorated: decorators (classdef | funcdef | async_funcdef) > async_funcdef: ASYNC funcdef > > async_stmt: ASYNC (funcdef | with_stmt) # will add for_stmt later > > compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt > | funcdef | classdef | decorated | async_stmt) > > atom: ('(' [yield_expr|await_expr|testlist_comp] ')' | > '[' [testlist_comp] ']' | > '{' [dictorsetmaker] '}' | > NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False?) > > expr_stmt: testlist_star_expr (augassign (yield_expr|await_expr|testlist) | > ('=' (yield_expr|await_expr|testlist_star_expr))*) > > > Transition Period Shortcomings > ------------------------------ > > There is just one. > > Until ``async`` and ``await`` are not proper keywords, it is not possible (or > at least very hard) to fix ``tokenizer.c`` to recognize them on the **same > line** with ``def`` keyword:: > > # async and await will always be parsed as variables > > async def outer(): # 1 > def nested(a=(await fut)): > pass > > async def foo(): return (await fut) # 2 > > Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, a > ``SyntaxError`` will be raised. > > To workaround these issues, the above examples can be easily rewritten to a > more readable form:: > > async def outer(): # 1 > a_default = await fut > def nested(a=a_default): > pass > > async def foo(): # 2 > return (await fut) > > This limitation will go away as soon as ``async`` and ``await`` ate proper > keywords. Or if it's decided to use a future import for this PEP. > > > Deprecation Plans > ----------------- > > ``async`` and ``await`` names will be softly deprecated in CPython 3.5 and 3.6. > In 3.7 we will transform them to proper keywords. Making ``async`` and > ``await`` proper keywords before 3.7 might make it harder for people to port > their code to Python 3. > > > asyncio > ------- > > ``asyncio`` module was adapted and tested to work with coroutines and new > statements. Backwards compatibility is 100% preserved. > > The required changes are mainly: > > 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.async_def()`` > function. > > 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. > > 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate > ``async()`` function. > > > Design Considerations > ===================== > > No implicit wrapping in Futures > ------------------------------- > > There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key > difference is that JavaScript "async functions" always return a Promise. While > this approach has some advantages, it also implies that a new Promise object is > created on each "async function" invocation. > > We could implement a similar functionality in Python, by wrapping all > coroutines in a Future object, but this has the following disadvantages: > > 1. Performance. A new Future object would be instantiated on each coroutine > call. Moreover, this makes implementation of ``await`` expressions slower > (disabling optimizations of ``yield from``). > > 2. A new built-in ``Future`` object would need to be added. > > 3. Coming up with a generic ``Future`` interface that is usable for any use > case in any framework is a very hard to solve problem. > > 4. It is not a feature that is used frequently, when most of the code is > coroutines. > > > Why "async" and "await" keywords > -------------------------------- > > async/await is not a new concept in programming languages: > > * C# has it since long time ago [5]_; > > * proposal to add async/await in ECMAScript 7 [2]_; > see also Traceur project [9]_; > > * Facebook's Hack/HHVM [6]_; > > * Google's Dart language [7]_; > > * Scala [8]_; > > * proposal to add async/await to C++ [10]_; > > * and many other less popular languages. > > This is a huge benefit, as some users already have experience with async/await, > and because it makes working with many languages in one project easier (Python > with ECMAScript 7 for instance). > > > Why "__aiter__" is a coroutine > ------------------------------ > > In principle, ``__aiter__`` could be a regular function. There are several > good reasons to make it a coroutine: > > * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` methods are > coroutines, users would often make a mistake defining it as ``async`` > anyways; > > * there might be a need to run some asynchronous operations in ``__aiter__``, > for instance to prepare DB queries or do some file operation. > > > Importance of "async" keyword > ----------------------------- > > While it is possible to just implement ``await`` expression and treat all > functions with at least one ``await`` as coroutines, this approach makes > APIs design, code refactoring and its long time support harder. > > Let's pretend that Python only has ``await`` keyword:: > > def useful(): > ... > await log(...) > ... > > def important(): > await useful() > > If ``useful()`` function is refactored and someone removes all ``await`` > expressions from it, it would become a regular python function, and all code > that depends on it, including ``important()`` would be broken. To mitigate this > issue a decorator similar to ``@asyncio.coroutine`` has to be introduced. > > > Why "async def" > --------------- > > For some people bare ``async name(): pass`` syntax might look more appealing > than ``async def name(): pass``. It is certainly easier to type. But on the > other hand, it breaks the symmetry between ``async def``, ``async with`` and > ``async for``, where ``async`` is a modifier, stating that the statement is > asynchronous. It is also more consistent with the existing grammar. > > > Why not a __future__ import > --------------------------- > > ``__future__`` imports are inconvenient and easy to forget to add. Also, they > are enabled for the whole source file. Consider that there is a big project > with a popular module named "async.py". With future imports it is required to > either import it using ``__import__()`` or ``importlib.import_module()`` calls, > or to rename the module. The proposed approach makes it possible to continue > using old code and modules without a hassle, while coming up with a migration > plan for future python versions. > > > Why magic methods start with "a" > -------------------------------- > > New asynchronous magic methods ``__aiter__``, ``__anext__``, ``__aenter__``, > and ``__aexit__`` all start with the same prefix "a". An alternative proposal > is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. > However, to align new magic methods with the existing ones, such as > ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > > > Why not reuse existing magic names > ---------------------------------- > > An alternative idea about new asynchronous iterators and context managers was > to reuse existing magic methods, by adding an ``async`` keyword to their > declarations:: > > class CM: > async def __enter__(self): # instead of __aenter__ > ... > > This approach has the following downsides: > > * it would not be possible to create an object that works in both ``with`` and > ``async with`` statements; > > * it would look confusing and would require some implicit magic behind the > scenes in the interpreter; > > * one of the main points of this proposal is to make coroutines as simple > and foolproof as possible. > > > Comprehensions > -------------- > > For the sake of restricting the broadness of this PEP there is no new syntax > for asynchronous comprehensions. This should be considered in a separate PEP, > if there is a strong demand for this feature. > > > Performance > =========== > > Overall Impact > -------------- > > This proposal introduces no observable performance impact. Here is an output > of python's official set of benchmarks [4]_: > > :: > > python perf.py -r -b default ../cpython/python.exe ../cpython-aw/python.exe > > [skipped] > > Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: > Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 > x86_64 i386 > > Total CPU cores: 8 > > ### etree_iterparse ### > Min: 0.365359 -> 0.349168: 1.05x faster > Avg: 0.396924 -> 0.379735: 1.05x faster > Significant (t=9.71) > Stddev: 0.01225 -> 0.01277: 1.0423x larger > > The following not significant results are hidden, use -v to show them: > django_v2, 2to3, etree_generate, etree_parse, etree_process, fastpickle, > fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. > > > Tokenizer modifications > ----------------------- > > There is no observable slowdown of parsing python files with the modified > tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated 1000 > times) takes the same amount of time. > > > async/await > ----------- > > The following micro-benchmark was used to determine performance difference > between "async" functions and generators:: > > import sys > import time > > def binary(n): > if n <= 0: > return 1 > l = yield from binary(n - 1) > r = yield from binary(n - 1) > return l + 1 + r > > async def abinary(n): > if n <= 0: > return 1 > l = await abinary(n - 1) > r = await abinary(n - 1) > return l + 1 + r > > def timeit(gen, depth, repeat): > t0 = time.time() > for _ in range(repeat): > list(gen(depth)) > t1 = time.time() > print('{}({}) * {}: total {:.3f}s'.format( > gen.__name__, depth, repeat, t1-t0)) > > The result is that there is no observable performance difference. Minimum > timing of 3 runs > > :: > > abinary(19) * 30: total 12.985s > binary(19) * 30: total 12.953s > > Note that depth of 19 means 1,048,575 calls. > > > Reference Implementation > ======================== > > The reference implementation can be found here: [3]_. > > List of high-level changes and new protocols > -------------------------------------------- > > 1. New syntax for defining coroutines: ``async def`` and new ``await`` > keyword. > > 2. New ``__await__`` method for Future-like objects. > > 3. New syntax for asynchronous context managers: ``async with``. And > associated protocol with ``__aenter__`` and ``__aexit__`` methods. > > 4. New syntax for asynchronous iteration: ``async for``. And associated > protocol with ``__aiter__``, ``__aexit__`` and new built-in exception > ``StopAsyncIteration``. > > 5. New AST nodes: ``AsyncFor``, ``AsyncWith``, ``Await``; ``FunctionDef`` AST > node got a new argument ``is_async``. > > 6. New functions: ``sys.set_async_wrapper(callback)`` and > ``types.async_def(gen)``. > > 7. New ``CO_ASYNC`` bit flag for code objects. > > While the list of changes and new things is not short, it is important to > understand, that most users will not use these features directly. It is > intended to be used in frameworks and libraries to provide users with > convenient to use and unambiguous APIs with ``async def``, ``await``, ``async > for`` and ``async with`` syntax. > > > Working example > --------------- > > All concepts proposed in this PEP are implemented [3]_ and can be tested. > > :: > > import asyncio > > > async def echo_server(): > print('Serving on localhost:8000') > await asyncio.start_server(handle_connection, 'localhost', 8000) > > > async def handle_connection(reader, writer): > print('New connection...') > > while True: > data = await reader.read(8192) > > if not data: > break > > print('Sending {:.10}... back'.format(repr(data))) > writer.write(data) > > > loop = asyncio.get_event_loop() > loop.run_until_complete(echo_server()) > try: > loop.run_forever() > finally: > loop.close() > > > References > ========== > > .. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine > > .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions > > .. [3] https://github.com/1st1/cpython/tree/await > > .. [4] https://hg.python.org/benchmarks > > .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx > > .. [6] http://docs.hhvm.com/manual/en/hack.async.php > > .. [7] https://www.dartlang.org/articles/await-async/ > > .. [8] http://docs.scala-lang.org/sips/pending/async.html > > .. [9] https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental > > .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) > > > Acknowledgments > =============== > > I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew Svetlov, > and ?ukasz Langa for their initial feedback. > > > Copyright > ========= > > This document has been placed in the public domain. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From breamoreboy at yahoo.co.uk Sat Apr 18 04:26:49 2015 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 18 Apr 2015 03:26:49 +0100 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: On 18/04/2015 02:31, Dan O'Reilly wrote: > > This is exciting! I expect this will eliminate some of the confusion I > see from new users of asyncio on StackOverflow. > > I'm curious about what this means for asyncio best practices from 3.5 > onward, though. Is the use of @asyncio.coroutine/yield from being > deprecated in asyncio once this feature is added? I guess anyone that > needs to write backwards compatible code would need to continue using > the old way, so perhaps not yet? At the very least the docs should > probably mention that the async/wait syntax is available/preferred for > code that only needs to support 3.5+. If you're going the deprecation > route, the docs should be updated to use async/await, and probably just > include a note stating that the old @asyncio.coroutine/yield from method > is required to support 3.4 and lower. > Would people please be kind enough to hold their horses? I'm getting the impression that this is expected in 3.5, but according to PEP 478 beta 1 is due on 24th May 2015, so it looks as if somebody has got to get their skates on. Or have I (as usual) missed something, in which case I'll apologise now, mainly because it's Saturday 18/04/2015 03:26 BST and I'm heading off to bed. Good night all :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From rosuav at gmail.com Sat Apr 18 04:52:15 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Apr 2015 12:52:15 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: On Sat, Apr 18, 2015 at 12:26 PM, Mark Lawrence wrote: > Would people please be kind enough to hold their horses? I'm getting the > impression that this is expected in 3.5, but according to PEP 478 beta 1 is > due on 24th May 2015, so it looks as if somebody has got to get their skates > on. Or have I (as usual) missed something, in which case I'll apologise > now, mainly because it's Saturday 18/04/2015 03:26 BST and I'm heading off > to bed. Good night all :) There is already a reference implementation, so it's possible it'll get there in time. That's for the release manager to decide, though. Even if it doesn't land in 3.5, all that'll change is that all the numbers go up by 0.1. It'll be fundamentally the same proposal. ChrisA From guido at python.org Sat Apr 18 05:26:50 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 17 Apr 2015 20:26:50 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: On Fri, Apr 17, 2015 at 7:52 PM, Chris Angelico wrote: > On Sat, Apr 18, 2015 at 12:26 PM, Mark Lawrence > wrote: > > Would people please be kind enough to hold their horses? I'm getting the > > impression that this is expected in 3.5, but according to PEP 478 beta 1 > is > > due on 24th May 2015, so it looks as if somebody has got to get their > skates > > on. Or have I (as usual) missed something, in which case I'll apologise > > now, mainly because it's Saturday 18/04/2015 03:26 BST and I'm heading > off > > to bed. Good night all :) > > There is already a reference implementation, so it's possible it'll > get there in time. That's for the release manager to decide, though. > > Even if it doesn't land in 3.5, all that'll change is that all the > numbers go up by 0.1. It'll be fundamentally the same proposal. > It's interesting to see the world of PEPs change. We used to wrangle for years over some topics. I hope we aren't going *too* fast, but I think 3.5 will be quite the release (even if this particular one gets delayed). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rustompmody at gmail.com Sat Apr 18 06:48:35 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Sat, 18 Apr 2015 10:18:35 +0530 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) Message-ID: On Sat, Apr 18, 2015 at 12:28 AM, Yury Selivanov wrote: > Hello python-ideas, > > Here's my proposal to add async/await in Python. > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, I am also interested in this topic --- from the other side. As a teacher of python it is my finding that the terminology/documentation around generators is rather chaotic and messy. Basically given: def foo(): yield 1 bar = foo() what do we call foo and what do we call bar? It is suggested that foo is "generator-function" and bar is "generator-object" Unfortunately python does not aid this distinction; witness >>> def foo(): ... yield 1 ... >>> bar = foo() >>> type(foo) >>> type(bar) >>> I asked about this on the python list http://code.activestate.com/lists/python-list/682286/ And it seems that many more dark corners emerged in the docs on this subject in that discussion. Should I start a separate thread? From yselivanov.ml at gmail.com Sat Apr 18 07:31:06 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 18 Apr 2015 01:31:06 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: <5531EC1A.6040609@gmail.com> Hi Dan, On 2015-04-17 9:31 PM, Dan O'Reilly wrote: > This is exciting! I expect this will eliminate some of the confusion I see > from new users of asyncio on StackOverflow. Thanks! I'm sure it will! > > I'm curious about what this means for asyncio best practices from 3.5 > onward, though. Is the use of @asyncio.coroutine/yield from being > deprecated in asyncio once this feature is added? I guess anyone that needs > to write backwards compatible code would need to continue using the old > way, so perhaps not yet? Definitely not yet. I expect libraries to start offering extended functionality (like nice context managers and iterators) for those who can use async/await, but to keep backwards compatibility until 3.6 at least. > At the very least the docs should probably mention > that the async/wait syntax is available/preferred for code that only needs > to support 3.5+. If you're going the deprecation route, the docs should be > updated to use async/await, and probably just include a note stating that > the old @asyncio.coroutine/yield from method is required to support 3.4 and > lower. This is something that we definitely have to consider if the PEP gets accepted. Best, Yury From yselivanov.ml at gmail.com Sat Apr 18 07:36:26 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 18 Apr 2015 01:36:26 -0400 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: References: Message-ID: <5531ED5A.6040108@gmail.com> Hi Rustom, On 2015-04-18 12:48 AM, Rustom Mody wrote: > On Sat, Apr 18, 2015 at 12:28 AM, Yury Selivanov > wrote: >> Hello python-ideas, >> >> Here's my proposal to add async/await in Python. >> >> I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > I am also interested in this topic --- from the other side. > As a teacher of python it is my finding that the > terminology/documentation around generators is rather chaotic and > messy. > Basically given: > def foo(): > yield 1 > bar = foo() > > what do we call foo and what do we call bar? > It is suggested that foo is "generator-function" and bar is "generator-object" Correct. > Unfortunately python does not aid this distinction; witness > >>>> def foo(): > ... yield 1 > ... >>>> bar = foo() >>>> type(foo) > >>>> type(bar) > > I asked about this on the python list > http://code.activestate.com/lists/python-list/682286/ There is a set of functions in the inspect module to help with runtime introspection: inspect.isfunction() inspect.isgenerator() inspect.isgeneratorfunction() > > And it seems that many more dark corners emerged in the docs on this > subject in that discussion. > > Should I start a separate thread? I think that yes, this has to be a different thread. Async/await thread is specifically to discuss PEP 492. Thanks, Yury From abarnert at yahoo.com Sat Apr 18 08:05:00 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 17 Apr 2015 23:05:00 -0700 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: References: Message-ID: On Apr 17, 2015, at 21:48, Rustom Mody wrote: > > On Sat, Apr 18, 2015 at 12:28 AM, Yury Selivanov > wrote: >> Hello python-ideas, >> >> Here's my proposal to add async/await in Python. >> >> I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > > I am also interested in this topic --- from the other side. > As a teacher of python it is my finding that the > terminology/documentation around generators is rather chaotic and > messy. > Basically given: > def foo(): > yield 1 > bar = foo() > > what do we call foo and what do we call bar? > It is suggested that foo is "generator-function" and bar is "generator-object" > Unfortunately python does not aid this distinction; witness > >>>> def foo(): > ... yield 1 > ... >>>> bar = foo() >>>> type(foo) > >>>> type(bar) > I assume you're ok with the type(bar); the type of a generator object is named "generator", just like the type of a bound method object is named "method" and the type of an integer object is named "int", and so on. So presumably you don't like the former. But what about it? The only thing I can imagine is the fact that, even though a generator function is conceptually a "subkind" of function, there is no generator function subtype of the function type? But so what? There's no closure function subtype, just functions whose closure is nonempty; there's no positive int subtype, just ints whose value is positive; etc. And likewise, there's no callable supertype that function and method (and other things) inherit. If there were some additional API that a generator function should provide that other functions don't (or some implementation reason that makes it worth subclassing for convenience, I suppose), that would be a good reason for needing a subtype. For example, generator a subtype of iterator, just as file is, because they both add new methods that don't make sense for the base iterator type. But just wanting repr(type(x)) to give you more information about x, that's not a good reason to add a subtype. So, it seems to me like both of these are returning something you should reasonably expect. Although you didn't bring this up here, I'll bet somewhere in that thread you point out that it's unfortunate that, in loose usage, people sometimes call a generator function--or, more often, the source code defining one--a generator, in the same way they sometimes call the source code defining a function a function, but there's ambiguity like that all over the natural language we use to talk about programming; as long as there are terms we can use when we need to be strict, it's rarely a problem. And I think we have such terms here: a generator function is the kind of function used to implement generator objects, while are a kind (and indeed a subtype) of iterator objects. Changing the strict names from the obvious ones to something that nobody can get wrong--like calling function definitions "code defs" so nobody loosely says "function" anymore--wouldn't be a win. Besides, the generator PEP and its discussion went over this in detail, and IIRC Guido's conclusion could be paraphrased as "I think this is FUD, but I don't think we'll be able to prove that except by looking at actual experience in a few years". In actual experience, I haven't seen anyone stumble over this except if they don't understand the basic generator abstraction at all. (I can't speak to whether that makes it harder to teach that abstraction to novices or not, but I would guess it only really affects the ones who are "reading ahead in class", who you can mostly count on to get things or ask good questions, because you can just avoid the loose usage entirely in class.) > I asked about this on the python list > http://code.activestate.com/lists/python-list/682286/ > > And it seems that many more dark corners emerged in the docs on this > subject in that discussion. > > Should I start a separate thread? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rustompmody at gmail.com Sat Apr 18 08:30:14 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Sat, 18 Apr 2015 12:00:14 +0530 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: References: Message-ID: Will write the points later but for now... On Sat, Apr 18, 2015 at 11:35 AM, Andrew Barnert wrote: > On Apr 17, 2015, at 21:48, Rustom Mody wrote: >> >> On Sat, Apr 18, 2015 at 12:28 AM, Yury Selivanov >> wrote: >>> Hello python-ideas, >>> >>> Here's my proposal to add async/await in Python. >>> >>> I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, >> >> I am also interested in this topic --- from the other side. >> As a teacher of python it is my finding that the >> terminology/documentation around generators is rather chaotic and >> messy. >> Basically given: >> def foo(): >> yield 1 >> bar = foo() >> >> what do we call foo and what do we call bar? >> It is suggested that foo is "generator-function" and bar is "generator-object" >> Unfortunately python does not aid this distinction; witness >> >>>>> def foo(): >> ... yield 1 >> ... >>>>> bar = foo() >>>>> type(foo) >> >>>>> type(bar) >> > > I assume you're ok with the type(bar); the type of a generator object is named "generator", just like the type of a bound method object is named "method" and the type of an integer object is named "int", and so on. > > So presumably you don't like the former. But what about it? > > The only thing I can imagine is the fact that, even though a generator function is conceptually a "subkind" of function, there is no generator function subtype of the function type? > > But so what? There's no closure function subtype, just functions whose closure is nonempty; there's no positive int subtype, just ints whose value is positive; etc. And likewise, there's no callable supertype that function and method (and other things) inherit. > > If there were some additional API that a generator function should provide that other functions don't (or some implementation reason that makes it worth subclassing for convenience, I suppose), that would be a good reason for needing a subtype. For example, generator a subtype of iterator, just as file is, because they both add new methods that don't make sense for the base iterator type. > > But just wanting repr(type(x)) to give you more information about x, that's not a good reason to add a subtype. > > So, it seems to me like both of these are returning something you should reasonably expect. > > Although you didn't bring this up here, I'll bet somewhere in that thread you point out that it's unfortunate that, in loose usage, people sometimes call a generator function Yes its this loose usage not just across the net but in the docs that in my experience are giving a great deal of trouble to noobs. Just to be clear: I am not talking about changing the language at all?; just streamlining the docs. I believe we just should take the current way of def-ining generators as a given. Then the discussions can be more meaningful and less FUD-ly I will write up a summary of that thread... ----------------- ? Well with the exception of areas where the line between semantics and docs is hazy eg help() and introspection From rustompmody at gmail.com Sat Apr 18 09:00:15 2015 From: rustompmody at gmail.com (Rustom Mody) Date: Sat, 18 Apr 2015 00:00:15 -0700 (PDT) Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: <5531ED5A.6040108@gmail.com> References: <5531ED5A.6040108@gmail.com> Message-ID: <68219140-b5b6-42f0-abcc-c9bf4cd1a2cf@googlegroups.com> On Saturday, April 18, 2015 at 11:07:24 AM UTC+5:30, Yury Selivanov wrote: > > Hi Rustom, > > On 2015-04-18 12:48 AM, Rustom Mody wrote: > > > Should I start a separate thread? > I think that yes, this has to be a different thread. > > Async/await thread is specifically to discuss PEP 492. > > This is confusing... I replied to your mail and changed the subject. Both gmail and googlegroups show a new thread created So how it is in other newsreaders/mail clients I dont know -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Apr 18 02:21:36 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 18 Apr 2015 12:21:36 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: <5531A390.8060405@canterbury.ac.nz> Yury Selivanov wrote: > > Here's my proposal to add async/await in Python. You've essentially reinvented PEP 3152 - Cofunctions. https://www.python.org/dev/peps/pep-3152/ Here's a summary of the relationships between them: PEP 3152 Nearest equivalent in Yury's PEP -------- -------------------------------- codef f(args): async def f(args): cocall f(args) await f(args) __cocall__ __await__ costart() async_def() There is currently no equivalent of "async for" and "async with" in PEP 3152, but they could easily be added. I would probably spell them "cofor" and "cowith". As the author of PEP 3152 I'm obviously biased, but I think my spellings are more elegant and less disruptive to reading of the code. PEP 3152 is currently marked as deferred. Maybe it's time to revive it? If Yury's pep is to be considered, we ought to discuss the relative merits of the two. -- Greg From steve at pearwood.info Sat Apr 18 12:26:34 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Apr 2015 20:26:34 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: <20150417211030.GE28035@stoneleaf.us> References: <20150417211030.GE28035@stoneleaf.us> Message-ID: <20150418102633.GV5663@ando.pearwood.info> On Fri, Apr 17, 2015 at 02:10:30PM -0700, Ethan Furman wrote: > On 04/18, Ionel Cristian M?rie? wrote: > > > Also there's the issue about not being able to implement a true proxy (as > > outlined before). > > Proxies are a bit of a pain. But you can create your own callable function. > > Something like (untested): > > def callable(obj): > try: > func = obj.__call__ > return True > except AttributeError: > return False You can define it like that, but it doesn't work. Using the above definition of callable: py> class Spam(object): ... pass ... py> x = Spam() py> x.__call__ = lambda self: 23 py> callable(x) True py> x() Traceback (most recent call last): File "", line 1, in TypeError: 'Spam' object is not callable "foo()" syntax does not look for a __call__ method on the instance, only on the type. The current behaviour of callable is correct: py> builtins.callable(x) False With new-style classes (but not classic classes), all dunder methods are only accessed through the class, not the instance. Hence type(obj).__call__ is correct and obj.__call__ is incorrect. Could this be changed? That deserves a new thread, at least, and possibly a PEP, but briefly: (1) The current behaviour is documented, so it would require some form of transition; (2) The current behaviour is intended as an optimization. Using Python 2.7 on my system, I find that a minimal __add__ method is more than three times faster using a new-style class compared to a classic class. -- Steve From toddrjen at gmail.com Sat Apr 18 12:41:01 2015 From: toddrjen at gmail.com (Todd) Date: Sat, 18 Apr 2015 12:41:01 +0200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: On Apr 17, 2015 8:59 PM, "Yury Selivanov" wrote: > > > > async with EXPR as VAR: > BLOCK > > > > > async for TARGET in ITER: > BLOCK > else: > BLOCK2 > Is this really something that should restricted to async? I could see similar syntax being useful for multithreading, multiprocessing, MPI, etc. What about a more general language feature, where any class implementing certain magic methods could be used in this way? -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Sat Apr 18 12:41:04 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 13:41:04 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: <20150418102633.GV5663@ando.pearwood.info> References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> Message-ID: On Sat, Apr 18, 2015 at 1:26 PM, Steven D'Aprano wrote: > > With new-style classes (but not classic classes), all dunder methods are > only accessed through the class, not the instance. Hence > type(obj).__call__ is correct and obj.__call__ is incorrect. ?Yes, this is correct. But unfortunately is also very commonly misunderstood, because even special methods will work through the descriptor protocol. The __new__ method is the only special case here (the infamous "special cased static method"). py> x.__call__ = lambda self: 23 > The issue never was about patching __call__ on an instance, it's about making `callable` respect how the method is actually looked up fully (lookup on type + descriptor protocol). What `callable` is missing now is an additional check that will run the descriptor protocol. ?Could this be changed? That deserves a new thread, at least, and > possibly a PEP, but briefly:? ?So what exactly are you proposing, making a PEP that documents the fact that functions are descriptors and the descriptor protocol is used even for special methods? To give more context here, this is valid and it works right now: ?>>> class NonIter: > ... pass > ... > >>> iter(NonIter()) > Traceback (most recent call last): > File "", line 1, in > TypeError: 'NonIter' object is not iterable > >>> > >>> class DynamicNonIter: > ... has_iter = False > ... > ... @property > ... def __iter__(self): > ... if self.has_iter: > ... from functools import partial > ... return partial(iter, [1, 2, 3]) > ... else: > ... raise AttributeError("Not really ...") > ... > >>> dni = DynamicNonIter() > >>> iter(dni) > Traceback (most recent call last): > File "", line 1, in > TypeError: 'DynamicNonIter' object is not iterable > >>> dni.has_iter = True > >>> iter(dni) > ?I don't see why `callable` shouldn't work the same.? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Apr 18 13:05:31 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Apr 2015 21:05:31 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: <20150418110531.GW5663@ando.pearwood.info> On Fri, Apr 17, 2015 at 09:04:56PM -0400, Terry Reedy wrote: > It works because CPython apparently implements c1 + c2, at least for > user class instances, as c1.__add__(c2), instead of the presumed > equivalent C.__add__(c1, c2). c1.__add__ is normally a bound method > with c1 and C.__add__ as attributes. c1.__add__(c2) then calls > C.__add__(c1, c2). I believe that analysis is wrong. In Python 3, and with new-style classes in Python 2, c1+c2 is implemented as type(c1).__add__(c1, c2) and not c1.__add__(c2). (Actually, it is more complex than that, since the + operator also has to consider __radd__, and various other complications. But let's ignore those complications.) We can test this by giving instances an __add__ attribute in the instance dict, and see what happens. First I confirm that classic classes do implement a+b as a.__add__(b): py> class Classic: ... def __add__(self, other): ... return 23 ... py> classic = Classic() py> classic + None 23 py> from types import MethodType py> classic.__add__ = MethodType(lambda self, other: 42, classic) py> classic + None 42 But the same is not the case with a new-style class: py> class Newstyle(object): ... def __add__(self, other): ... return 23 ... py> newstyle = Newstyle() py> newstyle + None 23 py> newstyle.__add__ = MethodType(lambda self, other: 42, newstyle) py> newstyle + None # ignores the __add__ method on the instance 23 py> newstyle.__add__(None) 42 This demonstrates that when it comes to newstyle classes, dunder methods on the instance are not used. I don't think that using __add__ as a property is intended, but it works, and I don't think it is an accident that it works. It is simply a consequence of how descriptors work in Python. Here is an example: class Demo(object): @property def __add__(self): # Note that there is only a self argument. # Return a function of one argument. If we use a closure, # we can access self. return lambda other: [self, other] py> x = Demo() py> x + None [<__main__.Demo object at 0xb7c2f5ec>, None] Let's break that process down. As I show above, + is implemented as type(x).__add__ which returns a property object, a descriptor. The descriptor protocol says that the __get__ method will be called. So we have: x + None -> looks up type(x).__add__ -> which calls __get__ -> and finally calls the result of that py> type(x).__add__ py> type(x).__add__.__get__(x, type(x)) at 0xb7c29a04> py> type(x).__add__.__get__(x, type(x))(None) [<__main__.Demo object at 0xb7c2f5ec>, None] -- Steve From steve at pearwood.info Sat Apr 18 13:23:32 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 18 Apr 2015 21:23:32 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> Message-ID: <20150418112332.GX5663@ando.pearwood.info> On Sat, Apr 18, 2015 at 01:41:04PM +0300, Ionel Cristian M?rie? wrote: > On Sat, Apr 18, 2015 at 1:26 PM, Steven D'Aprano > wrote: > > > > > With new-style classes (but not classic classes), all dunder methods are > > only accessed through the class, not the instance. Hence > > type(obj).__call__ is correct and obj.__call__ is incorrect. > > > ?Yes, this is correct. But unfortunately is also very commonly > misunderstood, because even special methods will work through the > descriptor protocol. The __new__ method is the only special case here (the > infamous "special cased static method"). I must admit that until now I had never even thought about making a property with a dunder name. But now that I have considered it, why should it not work? The property (or other descriptor) is on the class, not the instance. py> class Test(object): ... @property ... def __len__(self): ... return lambda: 42 ... py> x = Test() py> len(x) 42 py> type(x).__len__.__get__(x, type(x))() 42 But this won't work on the instance. > The issue never was about patching __call__ on an instance, it's about > making `callable` respect how the method is actually looked up fully > (lookup on type + descriptor protocol). What `callable` is missing now is > an additional check that will run the descriptor protocol. Okay, I misunderstood you, my apologies. I was mislead by your original suggestion to use hasattr(x, '__call__'), sorry about that. Are you sure this doesn't already work? It works for me in Python 3.3: py> class CallableTest(object): ... @property ... def __call__(self): ... def inner(arg1, arg2): ... return [self, "called with", arg1, arg2] ... return inner ... py> x = CallableTest() py> x(1, 2) [<__main__.CallableTest object at 0xb7b9072c>, 'called with', 1, 2] py> callable(x) True but perhaps I have missed something. > > ?Could this be changed? That deserves a new thread, at least, and > > possibly a PEP, but briefly:? > > > ?So what exactly are you proposing, making a PEP that documents the fact > that functions are descriptors and the descriptor protocol is used even for > special methods? No, I thought you wanted dunder methods to be used from the instance __dict__, like with classic classes. If that is not your intent, then my comments about a PEP are irrelevant. -- Steve From contact at ionelmc.ro Sat Apr 18 13:41:29 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 14:41:29 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: <20150418112332.GX5663@ando.pearwood.info> References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sat, Apr 18, 2015 at 2:23 PM, Steven D'Aprano wrote: > Are you sure this doesn't already work? It works for me in Python 3.3: It doesn't really work because of the incomplete checks done in `callable_builtin`. ?This is what I've tried: >>> class DynamicCallable: > ... is_callable = True > ... > ... def __init__(self, target): > ... self.target = target > ... > ... @property > ... def __call__(self): > ... if self.is_callable: > ... return self.target > ... else: > ... raise AttributeError("Not really ...") > ... > >>> dc = DynamicCallable(print) > >>> dc(1, 2, 3) > 1 2 3 > >>> callable(dc) > True > >>> dc.is_callable = False > >>> callable(dc) > True ###### This should be False :( > If the "bug" is fixed, then the last thing in the above example would return False. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 18 14:25:19 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 18 Apr 2015 22:25:19 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sat, Apr 18, 2015 at 9:41 PM, Ionel Cristian M?rie? wrote: >>>> class DynamicCallable: > ... is_callable = True > ... > ... def __init__(self, target): > ... self.target = target > ... > ... @property > ... def __call__(self): > ... if self.is_callable: > ... return self.target > ... else: > ... raise AttributeError("Not really ...") Before charging ahead and making the callable() function check for this, is it actually a supported concept? Is it intentional that property-getter functions can raise AttributeError to signal that the attribute does not (currently) exist? Because if that's a hack, then there's no reason to support it, and callable() is absolutely correct to say "there is a __call__ attribute on the class, ergo it's callable". ChrisA From jsbueno at python.org.br Sat Apr 18 14:42:41 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Sat, 18 Apr 2015 09:42:41 -0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: On 17 April 2015 at 18:39, Guido van Rossum wrote: > I think you've found an unintended and undocumented backdoor. I admit I > don't understand how this works in CPython. Overloaded operators like > __add__ or __call__ should be methods in the class, and we don't look for > them in the instance. But somehow defining them with @property works (I > guess because @property is in the class). > > What's different for __call__ is that callable() exists. And this is > probably why I exorcised it Python 3.0 -- but apparently it's back. :-( And for much that I've searched I've never found out the reasoning on that exorcism. Since you are at it, could you describe it? I am glad it is back - I think it is definitely needed - even if just works in a sort of naive way. From brett at python.org Sat Apr 18 15:23:07 2015 From: brett at python.org (Brett Cannon) Date: Sat, 18 Apr 2015 13:23:07 +0000 Subject: [Python-ideas] async/await in Python In-Reply-To: <55318FA5.6010403@gmail.com> References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> <55318FA5.6010403@gmail.com> Message-ID: +1 from me on the PEP! Very thorough and well argued. On Fri, Apr 17, 2015 at 6:57 PM Yury Selivanov wrote: > Chris, > On 2015-04-17 6:46 PM, Chris Angelico wrote: > > On Sat, Apr 18, 2015 at 8:18 AM, Yury Selivanov > wrote: > >> On 2015-04-17 6:00 PM, Chris Angelico wrote: > >>> On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov < > yselivanov.ml at gmail.com> > >>> wrote: > [SNIP] > > > > >> Let's see what python-ideas thinks about it. I'm fine if everybody > >> wants __future__ imports. I will only have to rollback my changes > >> in tokenizer.c and change few tokens in Grammar to make the > >> reference implementation work. > > In case it wasn't clear from my previous post, I'm +1 on using a > > __future__ import. Victor's idea of an optional directive seems > > interesting, but I'm not sure how useful it'd be in reality; does > > complicating the rules offer more benefit than simply having a keyword > > governed by __future__? > I'm OK with __future__ import. And I'll be extremely happy > if that's the only thing we'll be all discussing here ;-) > > Let's see how the discussion goes, and if everybody on the > list wants __future__ and Guido approves, I'll update > the PEP and ref implementation! > +1 for the __future__ statement. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Apr 18 15:40:36 2015 From: brett at python.org (Brett Cannon) Date: Sat, 18 Apr 2015 13:40:36 +0000 Subject: [Python-ideas] async/await in Python In-Reply-To: <5531A390.8060405@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> Message-ID: On Sat, Apr 18, 2015 at 5:53 AM Greg Ewing wrote: > Yury Selivanov wrote: > > > > Here's my proposal to add async/await in Python. > > You've essentially reinvented PEP 3152 - Cofunctions. > > https://www.python.org/dev/peps/pep-3152/ > > Here's a summary of the relationships between them: > > PEP 3152 Nearest equivalent in Yury's PEP > -------- -------------------------------- > > codef f(args): async def f(args): > > cocall f(args) await f(args) > > __cocall__ __await__ > > costart() async_def() > > There is currently no equivalent of "async for" and > "async with" in PEP 3152, but they could easily be added. > I would probably spell them "cofor" and "cowith". > > As the author of PEP 3152 I'm obviously biased, but I > think my spellings are more elegant and less disruptive > to reading of the code. > I prefer Yury's spellings. The use of 'async' seems more obvious to me for what the term(s) are meant to convey. It also has the side-effect of being more obvious to people coming from other programming languages (I also asked my wife what she thought A-S-Y-N-C meant and she figured out that it meant "asynchronous" and she isn't a programmer). My brain also keeps trying to make actual words out of "cofor" and "cowith" unsuccessfully and just settles on gibberish and something to do with cows, respectively. > PEP 3152 is currently marked as deferred. Maybe it's > time to revive it? If Yury's pep is to be considered, > we ought to discuss the relative merits of the two. > Discussing differences in semantics makes sense if there are any. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sat Apr 18 17:17:47 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 18 Apr 2015 11:17:47 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <5531A390.8060405@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> Message-ID: Greg, thank you for raising PEP 3152. When I discussed PEP for async/await with Yuri and Guido offline at PyCon I've mentioned Cofunctions PEP. Sure, Yuri have remembered it on working on your own PEP. >From my perspective async/await syntax is better for teaching newcomers. cofor and cowith looks ugly and cryptic, sorry. There is simple mnemonic: rule async is for asynchronous. co- for coroutine forces you to describe what is coroutine first. Also cofunc, codef, cofor and cowith requires more keywords in python syntax, which is not our desire I guess. Anyway, thank for your PEP -- it gives an inspiration. Perhaps Yuri should mention it in own PEP 492 at least. On Fri, Apr 17, 2015 at 8:21 PM, Greg Ewing wrote: > Yury Selivanov wrote: >> >> >> Here's my proposal to add async/await in Python. > > > You've essentially reinvented PEP 3152 - Cofunctions. > > https://www.python.org/dev/peps/pep-3152/ > > Here's a summary of the relationships between them: > > PEP 3152 Nearest equivalent in Yury's PEP > -------- -------------------------------- > > codef f(args): async def f(args): > > cocall f(args) await f(args) > > __cocall__ __await__ > > costart() async_def() > > There is currently no equivalent of "async for" and > "async with" in PEP 3152, but they could easily be added. > I would probably spell them "cofor" and "cowith". > > As the author of PEP 3152 I'm obviously biased, but I > think my spellings are more elegant and less disruptive > to reading of the code. > > PEP 3152 is currently marked as deferred. Maybe it's > time to revive it? If Yury's pep is to be considered, > we ought to discuss the relative merits of the two. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov From jim.baker at python.org Sat Apr 18 18:01:04 2015 From: jim.baker at python.org (Jim Baker) Date: Sat, 18 Apr 2015 10:01:04 -0600 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: +1 Such support would make async coroutines work well on implementations like Jython, as opposed to the current situation that there's a large body of current async code, using the greenlet approach, that currently does not. Note that efficient support would currently require linking a jar from https://github.com/puniverse/quasar, but there's also the possibility that will be in a future Java as well (it's certainly discussed at most JVM Language Summits). Compare this proposal to what we have with greenlet and its use of a function named "switch", which cannot be statically determined to be such a thing. Without the static determination we get with keyword support, we cannot mark code as being a coroutine for Quasar, so it must be assumed to be so, at extra runtime overhead; also such support also would require that *all* Python code runs as Python bytecode (with the extra overhead that entails of running a Python bytecode VM on the JVM) since it could possibly be asynchronous. The alternative is to map such async coroutines to threads, much as I did for greenlet support with this proof-of-concept https://github.com/jythontools/artificialturf Although one can potentially run 1000s and possibly 10000s of threads on a JVM on say Linux, this is certainly not the case for all OS targets. On Fri, Apr 17, 2015 at 12:58 PM, Yury Selivanov wrote: > Why "async" and "await" keywords > -------------------------------- > > async/await is not a new concept in programming languages: > > * C# has it since long time ago [5]_; > > * proposal to add async/await in ECMAScript 7 [2]_; > see also Traceur project [9]_; > > * Facebook's Hack/HHVM [6]_; > > * Google's Dart language [7]_; > > * Scala [8]_; > > * proposal to add async/await to C++ [10]_; > > * and many other less popular languages. > > This is a huge benefit, as some users already have experience with > async/await, > and because it makes working with many languages in one project easier > (Python > with ECMAScript 7 for instance). > -- - Jim jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com} twitter.com/jimbaker github.com/jimbaker bitbucket.com/jimbaker -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Sat Apr 18 18:33:55 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sat, 18 Apr 2015 19:33:55 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sat, Apr 18, 2015 at 3:25 PM, Chris Angelico wrote: > Is it intentional that > property-getter functions can raise AttributeError to signal that the > attribute does not (currently) exist? Because if that's a hack, then > there's no reason to support it, and callable() is absolutely correct > to say "there is a __call__ attribute on the class, ergo it's > callable". > ?It's hard to say what's intended or not. At best is speculation, as python-the-language does not have a specification. What we have now is made up from many contributions from many people. What we should talk about is "do we want to have this intent or not". Now that's a worthwhile discussion?, that can avoid speculation and subjectivity about what's in the docs, what's a hack in contrast to what is in the docs, what's and if there's prior use. You can't talk about how tall your house is going to be until you know what you build it on - is it sand or is it rock? >From my perspective consistency is very important, and it's lacking here w.r.t. how `callable` behaves versus `hasattr` or the call operator. The `iter` builtin already handles `__iter__` differently compared to how `callable` handles `__call__` so there, what do we do with that then? Do we make `iter` raise AttributeError or TypeError (as it's now)? I strongly believe fixing `callable` is the shortest way to achieve some consistency here. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmludo at gmail.com Fri Apr 17 23:53:26 2015 From: gmludo at gmail.com (Ludovic Gasc) Date: Fri, 17 Apr 2015 23:53:26 +0200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: As a lambda end-user of AsyncIO, it seems promising. As I said to Andrew Svetlov during PyCON 2015, I really prefer this new syntax. For a new comer, it will be more easier to handle because you don't need to explain the first time what is a decorator, a yield, and a yield from in the same time that Futures, Coroutines and Tasks. It's a (small) distraction about Python internal plumber about AsyncIO implementation, instead to concentrate on async business logic implementation. Nevertheless, the only small fear I have is a potential "schism" in AsyncIO community like we have with Python 2/3. For end-products source code, no problems: For new projects you can use the new syntax, or migrate easily old projects, thanks to compatibility plan. But for open source libraries, you must keep old syntax is you want Python 3.3+ compatibility. I know it's a small change compare to Python 3 changes, and AsyncIO community is enough small for now and almost AsyncIO users are early adopters for now: It means it will be more fluid for the shift. Because async keyword integration will change the end-user source code, I vote for an integration in Python 3.5 (not 3.6): it will reduce the waiting time for the end-users to use that. In an ideal world, it should be useful if we could also use that with Python 3.3 and 3.4 to help migration, because mainstream Linux distributions for production (Debian Jessie, Ubuntu 14.04 and CentOS/RHEL 7) have packages for Python 3.4, not Python 3.5, and it will change only for the new versions of theses distributions. Nevertheless, I understand the technical and political reasons that it's impossible to do that: It's an internal change of CPython, can't be added via a third library. Hopefully, it's really easy to compile CPython, an AsyncIO early adopter can't be afraid to do that ;-) -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ 2015-04-17 20:58 GMT+02:00 Yury Selivanov : > Hello python-ideas, > > Here's my proposal to add async/await in Python. > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > and I hope that this PEP can be another big step. Who knows, maybe it > will be one of the reasons to drive people's interest towards Python 3. > > > PEP: XXX > Title: Coroutines with async and await syntax > Version: $Revision$ > Last-Modified: $Date$ > Author: Yury Selivanov > Discussions-To: Python-Dev > Python-Version: 3.5 > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 09-Apr-2015 > Post-History: > Resolution: > > > Abstract > ======== > > This PEP introduces new syntax for coroutines, asynchronous ``with`` > statements and ``for`` loops. The main motivation behind this proposal is > to > streamline writing and maintaining asynchronous code, as well as to > simplify > previously hard to implement code patterns. > > > Rationale and Goals > =================== > > Current Python supports implementing coroutines via generators (PEP 342), > further enhanced by the ``yield from`` syntax introduced in PEP 380. > This approach has a number of shortcomings: > > * it is easy to confuse coroutines with regular generators, since they > share > the same syntax; async libraries often attempt to alleviate this by using > decorators (e.g. ``@asyncio.coroutine`` [1]_); > > * it is not possible to natively define a coroutine which has no ``yield`` > or ``yield from`` statements, again requiring the use of decorators to > fix potential refactoring issues; > > * support for asynchronous calls is limited to expressions where ``yield`` > is > allowed syntactically, limiting the usefulness of syntactic features, > such > as ``with`` and ``for`` statements. > > This proposal makes coroutines a native Python language feature, and > clearly > separates them from generators. This removes generator/coroutine > ambiguity, > and makes it possible to reliably define coroutines without reliance on a > specific library. This also enables linters and IDEs to improve static > code > analysis and refactoring. > > Native coroutines and the associated new syntax features make it possible > to define context manager and iteration protocols in asynchronous terms. > As shown later in this proposal, the new ``async with`` statement lets > Python > programs perform asynchronous calls when entering and exiting a runtime > context, and the new ``async for`` statement makes it possible to perform > asynchronous calls in iterators. > > > Specification > ============= > > This proposal introduces new syntax and semantics to enhance coroutine > support > in Python, it does not change the internal implementation of coroutines, > which > are still based on generators. > > It is strongly suggested that the reader understands how coroutines are > implemented in Python (PEP 342 and PEP 380). It is also recommended to > read > PEP 3156 (asyncio framework). > > From this point in this document we use the word *coroutine* to refer to > functions declared using the new syntax. *generator-based coroutine* is > used > where necessary to refer to coroutines that are based on generator syntax. > > > New Coroutine Declaration Syntax > -------------------------------- > > The following new syntax is used to declare a coroutine:: > > async def read_data(db): > pass > > Key properties of coroutines: > > * Coroutines are always generators, even if they do not contain ``await`` > expressions. > > * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions > in > an ``async`` function. > > * Internally, a new code object flag - ``CO_ASYNC`` - is introduced to > enable > runtime detection of coroutines (and migrating existing code). > All coroutines have both ``CO_ASYNC`` and ``CO_GENERATOR`` flags set. > > * Regular generators, when called, return a *generator object*; similarly, > coroutines return a *coroutine object*. > > * ``StopIteration`` exceptions are not propagated out of coroutines, and > are > replaced with a ``RuntimeError``. For regular generators such behavior > requires a future import (see PEP 479). > > > types.async_def() > ----------------- > > A new function ``async_def(gen)`` is added to the ``types`` module. It > applies ``CO_ASYNC`` flag to the passed generator's code object, so that it > returns a *coroutine object* when called. > > This feature enables an easy upgrade path for existing libraries. > > > Await Expression > ---------------- > > The following new ``await`` expression is used to obtain a result of > coroutine > execution:: > > async def read_data(db): > data = await db.fetch('SELECT ...') > ... > > ``await``, similarly to ``yield from``, suspends execution of ``read_data`` > coroutine until ``db.fetch`` *awaitable* completes and returns the result > data. > > It uses the ``yield from`` implementation with an extra step of validating > its > argument. ``await`` only accepts an *awaitable*, which can be one of: > > * A *coroutine object* returned from a coroutine or a generator decorated > with > ``types.async_def()``. > > * An object with an ``__await__`` method returning an iterator. > > Any ``yield from`` chain of calls ends with a ``yield``. This is a > fundamental mechanism of how *Futures* are implemented. Since, > internally, > coroutines are a special kind of generators, every ``await`` is > suspended by > a ``yield`` somewhere down the chain of ``await`` calls (please refer to > PEP > 3156 for a detailed explanation.) > > To enable this behavior for coroutines, a new magic method called > ``__await__`` is added. In asyncio, for instance, to enable Future > objects > in ``await`` statements, the only change is to add ``__await__ = > __iter__`` > line to ``asyncio.Future`` class. > > Objects with ``__await__`` method are called *Future-like* objects in the > rest of this PEP. > > Also, please note that ``__aiter__`` method (see its definition below) > cannot > be used for this purpose. It is a different protocol, and would be like > using ``__iter__`` instead of ``__call__`` for regular callables. > > It is a ``SyntaxError`` to use ``await`` outside of a coroutine. > > > Asynchronous Context Managers and "async with" > ---------------------------------------------- > > An *asynchronous context manager* is a context manager that is able to > suspend > execution in its *enter* and *exit* methods. > > To make this possible, a new protocol for asynchronous context managers is > proposed. Two new magic methods are added: ``__aenter__`` and > ``__aexit__``. > Both must return an *awaitable*. > > An example of an asynchronous context manager:: > > class AsyncContextManager: > async def __aenter__(self): > await log('entering context') > > async def __aexit__(self, exc_type, exc, tb): > await log('exiting context') > > > New Syntax > '''''''''' > > A new statement for asynchronous context managers is proposed:: > > async with EXPR as VAR: > BLOCK > > > which is semantically equivalent to:: > > mgr = (EXPR) > aexit = type(mgr).__aexit__ > aenter = type(mgr).__aenter__(mgr) > exc = True > > try: > try: > VAR = await aenter > BLOCK > except: > exc = False > exit_res = await aexit(mgr, *sys.exc_info()) > if not exit_res: > raise > > finally: > if exc: > await aexit(mgr, None, None, None) > > > As with regular ``with`` statements, it is possible to specify multiple > context > managers in a single ``async with`` statement. > > It is an error to pass a regular context manager without ``__aenter__`` and > ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use > ``async with`` outside of a coroutine. > > > Example > ''''''' > > With asynchronous context managers it is easy to implement proper database > transaction managers for coroutines:: > > async def commit(session, data): > ... > > async with session.transaction(): > ... > await session.update(data) > ... > > Code that needs locking also looks lighter:: > > async with lock: > ... > > instead of:: > > with (yield from lock): > ... > > > Asynchronous Iterators and "async for" > -------------------------------------- > > An *asynchronous iterable* is able to call asynchronous code in its *iter* > implementation, and *asynchronous iterator* can call asynchronous code in > its > *next* method. To support asynchronous iteration: > > 1. An object must implement an ``__aiter__`` method returning an > *awaitable* > resulting in an *asynchronous iterator object*. > > 2. An *asynchronous iterator object* must implement an ``__anext__`` method > returning an *awaitable*. > > 3. To stop iteration```__anext__`` must raise a ``StopAsyncIteration`` > exception. > > An example of asynchronous iterable:: > > class AsyncIterable: > async def __aiter__(self): > return self > > async def __anext__(self): > data = await self.fetch_data() > if data: > return data > else: > raise StopAsyncIteration > > async def fetch_data(self): > ... > > > New Syntax > '''''''''' > > A new statement for iterating through asynchronous iterators is proposed:: > > async for TARGET in ITER: > BLOCK > else: > BLOCK2 > > which is semantically equivalent to:: > > iter = (ITER) > iter = await type(iter).__aiter__(iter) > running = True > while running: > try: > TARGET = await type(iter).__anext__(iter) > except StopAsyncIteration: > running = False > else: > BLOCK > else: > BLOCK2 > > > It is an error to pass a regular iterable without ``__aiter__`` method to > ``async for``. It is a ``SyntaxError`` to use ``async for`` outside of a > coroutine. > > As for with regular ``for`` statement, ``async for`` has an optional > ``else`` > clause. > > > Example 1 > ''''''''' > > With asynchronous iteration protocol it is possible to asynchronously > buffer > data during iteration:: > > async for data in cursor: > ... > > Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows > of data from a database after every ``N`` iterations. > > The following code illustrates new asynchronous iteration protocol:: > > class Cursor: > def __init__(self): > self.buffer = collections.deque() > > def _prefetch(self): > ... > > async def __aiter__(self): > return self > > async def __anext__(self): > if not self.buffer: > self.buffer = await self._prefetch() > if not self.buffer: > raise StopAsyncIteration > return self.buffer.popleft() > > then the ``Cursor`` class can be used as follows:: > > async for row in Cursor(): > print(row) > > which would be equivalent to the following code:: > > i = await Cursor().__aiter__() > while True: > try: > row = await i.__anext__() > except StopAsyncIteration: > break > else: > print(row) > > > Example 2 > ''''''''' > > The following is a utility class that transforms a regular iterable to an > asynchronous one. While this is not a very useful thing to do, the code > illustrates the relationship between regular and asynchronous iterators. > > :: > > class AsyncIteratorWrapper: > def __init__(self, obj): > self._it = iter(obj) > > async def __aiter__(self): > return self > > async def __anext__(self): > try: > value = next(self._it) > except StopIteration: > raise StopAsyncIteration > return value > > data = "abc" > it = AsyncIteratorWrapper("abc") > async for item in it: > print(it) > > > Why StopAsyncIteration? > ''''''''''''''''''''''' > > Coroutines are still based on generators internally. So, before PEP 479, > there > was no fundamental difference between > > :: > > def g1(): > yield from fut > return 'spam' > > and > > :: > > def g2(): > yield from fut > raise StopIteration('spam') > > And since PEP 479 is accepted and enabled by default for coroutines, the > following example will have its ``StopIteration`` wrapped into a > ``RuntimeError`` > > :: > > async def a1(): > await fut > raise StopIteration('spam') > > The only way to tell the outside code that the iteration has ended is to > raise > something other than ``StopIteration``. Therefore, a new built-in > exception > class ``StopAsyncIteration`` was added. > > Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions > raised > in coroutines are wrapped in ``RuntimeError``. > > > Debugging Features > ------------------ > > One of the most frequent mistakes that people make when using generators as > coroutines is forgetting to use ``yield from``:: > > @asyncio.coroutine > def useful(): > asyncio.sleep(1) # this will do noting without 'yield from' > > For debugging this kind of mistakes there is a special debug mode in > asyncio, > in which ``@coroutine`` decorator wraps all functions with a special object > with a destructor logging a warning. Whenever a wrapped generator gets > garbage > collected, a detailed logging message is generated with information about > where > exactly the decorator function was defined, stack trace of where it was > collected, etc. Wrapper object also provides a convenient ``__repr__`` > function with detailed information about the generator. > > The only problem is how to enable these debug capabilities. Since debug > facilities should be a no-op in production mode, ``@coroutine`` decorator > makes > the decision of whether to wrap or not to wrap based on an OS environment > variable ``PYTHONASYNCIODEBUG``. This way it is possible to run asyncio > programs with asyncio's own functions instrumented. > ``EventLoop.set_debug``, a > different debug facility, has no impact on ``@coroutine`` decorator's > behavior. > > With this proposal, coroutines is a native, distinct from generators, > concept. A new method ``set_async_wrapper`` is added to the ``sys`` > module, > with which frameworks can provide advanced debugging facilities. > > It is also important to make coroutines as fast and efficient as possible, > therefore there are no debug features enabled by default. > > Example:: > > async def debug_me(): > await asyncio.sleep(1) > > def async_debug_wrap(generator): > return asyncio.AsyncDebugWrapper(generator) > > sys.set_async_wrapper(async_debug_wrap) > > debug_me() # <- this line will likely GC the coroutine object and > # trigger AsyncDebugWrapper's code. > > assert isinstance(debug_me(), AsyncDebugWrapper) > > sys.set_async_wrapper(None) # <- this unsets any previously set > wrapper > assert not isinstance(debug_me(), AsyncDebugWrapper) > > If ``sys.set_async_wrapper()`` is called twice, the new wrapper replaces > the > previous wrapper. ``sys.set_async_wrapper(None)`` unsets the wrapper. > > > Glossary > ======== > > :Coroutine: > A coroutine function, or just "coroutine", is declared with ``async > def``. > It uses ``await`` and ``return value``; see `New Coroutine Declaration > Syntax`_ for details. > > :Coroutine object: > Returned from a coroutine function. See `Await Expression`_ for > details. > > :Future-like object: > An object with an ``__await__`` method. It is consumed by ``await`` > in a > coroutine. A coroutine waiting for a Future-like object is suspended > until > the Future-like object's ``__await__`` completes. ``await`` returns > the > result of the Future-like object. See `Await Expression`_ for details. > > :Awaitable: > A *future-like* object or a *coroutine object*. See `Await > Expression`_ > for details. > > :Generator-based coroutine: > Coroutines based in generator syntax. Most common example is > ``@asyncio.coroutine``. > > :Asynchronous context manager: > An asynchronous context manager has ``__aenter__`` and ``__aexit__`` > methods > and can be used with ``async with``. See > `Asynchronous Context Managers and "async with"`_ for details. > > :Asynchronous iterable: > An object with an ``__aiter__`` method, which must return an > *asynchronous > iterator* object. Can be used with ``async for``. See > `Asynchronous Iterators and "async for"`_ for details. > > :Asynchronous iterator: > An asynchronous iterator has an ``__anext__`` method.See > `Asynchronous Iterators and "async for"`_ for details. > > > List of functions and methods > ============================= > > ================= ======================================= ================= > Method Can contain Can't contain > ================= ======================================= ================= > async def func await, return value yield, yield > from > async def __a*__ await, return value yield, yield > from > def __a*__ return Future-like await > def __await__ yield, yield from, return iterable await > generator yield, yield from, return value await > ================= ======================================= ================= > > Where: > > * ""async def func": coroutine; > > * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, > ``__aexit__`` defined with the ``async`` keyword; > > * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` > defined without the ``async`` keyword, must return an *awaitable*; > > * "def __await__": ``__await__`` method to implement *Future-like* objects; > > * generator: a "regular" generator, function defined with ``def`` and which > contains a least one ``yield`` or ``yield from`` expression. > > *Future-like* is an object with an ``__await__`` method, see > `Await Expression`_ section for details. > > > Transition Plan > =============== > > To avoid backwards compatibility issues with ``async`` and ``await`` > keywords, > it was decided to modify ``tokenizer.c`` in such a way, that it: > > * recognizes ``async def`` name tokens combination (start of a coroutine); > > * keeps track of regular functions and coroutines; > > * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with > ``AWAIT`` when in the process of yielding tokens for coroutines. > > This approach allows for seamless combination of new syntax features (all > of > them available only in ``async`` functions) with any existing code. > > An example of having "async def" and "async" attribute in one piece of > code:: > > class Spam: > async = 42 > > async def ham(): > print(getattr(Spam, 'async')) > > # The coroutine can be executed and will print '42' > > > Backwards Compatibility > ----------------------- > > The only backwards incompatible change is an extra argument ``is_async`` to > ``FunctionDef`` AST node. But since it is a documented fact that the > structure > of AST nodes is an implementation detail and subject to change, this > should not > be considered a serious issue. > > > Grammar Updates > --------------- > > Grammar changes are also fairly minimal:: > > await_expr: AWAIT test > await_stmt: await_expr > > decorated: decorators (classdef | funcdef | async_funcdef) > async_funcdef: ASYNC funcdef > > async_stmt: ASYNC (funcdef | with_stmt) # will add for_stmt later > > compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt > | funcdef | classdef | decorated | async_stmt) > > atom: ('(' [yield_expr|await_expr|testlist_comp] ')' | > '[' [testlist_comp] ']' | > '{' [dictorsetmaker] '}' | > NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False?) > > expr_stmt: testlist_star_expr (augassign > (yield_expr|await_expr|testlist) | > ('=' (yield_expr|await_expr|testlist_star_expr))*) > > > Transition Period Shortcomings > ------------------------------ > > There is just one. > > Until ``async`` and ``await`` are not proper keywords, it is not possible > (or > at least very hard) to fix ``tokenizer.c`` to recognize them on the **same > line** with ``def`` keyword:: > > # async and await will always be parsed as variables > > async def outer(): # 1 > def nested(a=(await fut)): > pass > > async def foo(): return (await fut) # 2 > > Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, > a > ``SyntaxError`` will be raised. > > To workaround these issues, the above examples can be easily rewritten to a > more readable form:: > > async def outer(): # 1 > a_default = await fut > def nested(a=a_default): > pass > > async def foo(): # 2 > return (await fut) > > This limitation will go away as soon as ``async`` and ``await`` ate proper > keywords. Or if it's decided to use a future import for this PEP. > > > Deprecation Plans > ----------------- > > ``async`` and ``await`` names will be softly deprecated in CPython 3.5 and > 3.6. > In 3.7 we will transform them to proper keywords. Making ``async`` and > ``await`` proper keywords before 3.7 might make it harder for people to > port > their code to Python 3. > > > asyncio > ------- > > ``asyncio`` module was adapted and tested to work with coroutines and new > statements. Backwards compatibility is 100% preserved. > > The required changes are mainly: > > 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.async_def()`` > function. > > 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. > > 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate > ``async()`` function. > > > Design Considerations > ===================== > > No implicit wrapping in Futures > ------------------------------- > > There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key > difference is that JavaScript "async functions" always return a Promise. > While > this approach has some advantages, it also implies that a new Promise > object is > created on each "async function" invocation. > > We could implement a similar functionality in Python, by wrapping all > coroutines in a Future object, but this has the following disadvantages: > > 1. Performance. A new Future object would be instantiated on each > coroutine > call. Moreover, this makes implementation of ``await`` expressions > slower > (disabling optimizations of ``yield from``). > > 2. A new built-in ``Future`` object would need to be added. > > 3. Coming up with a generic ``Future`` interface that is usable for any use > case in any framework is a very hard to solve problem. > > 4. It is not a feature that is used frequently, when most of the code is > coroutines. > > > Why "async" and "await" keywords > -------------------------------- > > async/await is not a new concept in programming languages: > > * C# has it since long time ago [5]_; > > * proposal to add async/await in ECMAScript 7 [2]_; > see also Traceur project [9]_; > > * Facebook's Hack/HHVM [6]_; > > * Google's Dart language [7]_; > > * Scala [8]_; > > * proposal to add async/await to C++ [10]_; > > * and many other less popular languages. > > This is a huge benefit, as some users already have experience with > async/await, > and because it makes working with many languages in one project easier > (Python > with ECMAScript 7 for instance). > > > Why "__aiter__" is a coroutine > ------------------------------ > > In principle, ``__aiter__`` could be a regular function. There are several > good reasons to make it a coroutine: > > * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` methods > are > coroutines, users would often make a mistake defining it as ``async`` > anyways; > > * there might be a need to run some asynchronous operations in > ``__aiter__``, > for instance to prepare DB queries or do some file operation. > > > Importance of "async" keyword > ----------------------------- > > While it is possible to just implement ``await`` expression and treat all > functions with at least one ``await`` as coroutines, this approach makes > APIs design, code refactoring and its long time support harder. > > Let's pretend that Python only has ``await`` keyword:: > > def useful(): > ... > await log(...) > ... > > def important(): > await useful() > > If ``useful()`` function is refactored and someone removes all ``await`` > expressions from it, it would become a regular python function, and all > code > that depends on it, including ``important()`` would be broken. To > mitigate this > issue a decorator similar to ``@asyncio.coroutine`` has to be introduced. > > > Why "async def" > --------------- > > For some people bare ``async name(): pass`` syntax might look more > appealing > than ``async def name(): pass``. It is certainly easier to type. But on > the > other hand, it breaks the symmetry between ``async def``, ``async with`` > and > ``async for``, where ``async`` is a modifier, stating that the statement is > asynchronous. It is also more consistent with the existing grammar. > > > Why not a __future__ import > --------------------------- > > ``__future__`` imports are inconvenient and easy to forget to add. Also, > they > are enabled for the whole source file. Consider that there is a big > project > with a popular module named "async.py". With future imports it is > required to > either import it using ``__import__()`` or ``importlib.import_module()`` > calls, > or to rename the module. The proposed approach makes it possible to > continue > using old code and modules without a hassle, while coming up with a > migration > plan for future python versions. > > > Why magic methods start with "a" > -------------------------------- > > New asynchronous magic methods ``__aiter__``, ``__anext__``, > ``__aenter__``, > and ``__aexit__`` all start with the same prefix "a". An alternative > proposal > is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. > However, to align new magic methods with the existing ones, such as > ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > > > Why not reuse existing magic names > ---------------------------------- > > An alternative idea about new asynchronous iterators and context managers > was > to reuse existing magic methods, by adding an ``async`` keyword to their > declarations:: > > class CM: > async def __enter__(self): # instead of __aenter__ > ... > > This approach has the following downsides: > > * it would not be possible to create an object that works in both ``with`` > and > ``async with`` statements; > > * it would look confusing and would require some implicit magic behind the > scenes in the interpreter; > > * one of the main points of this proposal is to make coroutines as simple > and foolproof as possible. > > > Comprehensions > -------------- > > For the sake of restricting the broadness of this PEP there is no new > syntax > for asynchronous comprehensions. This should be considered in a separate > PEP, > if there is a strong demand for this feature. > > > Performance > =========== > > Overall Impact > -------------- > > This proposal introduces no observable performance impact. Here is an > output > of python's official set of benchmarks [4]_: > > :: > > python perf.py -r -b default ../cpython/python.exe > ../cpython-aw/python.exe > > [skipped] > > Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: > Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 > x86_64 i386 > > Total CPU cores: 8 > > ### etree_iterparse ### > Min: 0.365359 -> 0.349168: 1.05x faster > Avg: 0.396924 -> 0.379735: 1.05x faster > Significant (t=9.71) > Stddev: 0.01225 -> 0.01277: 1.0423x larger > > The following not significant results are hidden, use -v to show them: > django_v2, 2to3, etree_generate, etree_parse, etree_process, > fastpickle, > fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. > > > Tokenizer modifications > ----------------------- > > There is no observable slowdown of parsing python files with the modified > tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated > 1000 > times) takes the same amount of time. > > > async/await > ----------- > > The following micro-benchmark was used to determine performance difference > between "async" functions and generators:: > > import sys > import time > > def binary(n): > if n <= 0: > return 1 > l = yield from binary(n - 1) > r = yield from binary(n - 1) > return l + 1 + r > > async def abinary(n): > if n <= 0: > return 1 > l = await abinary(n - 1) > r = await abinary(n - 1) > return l + 1 + r > > def timeit(gen, depth, repeat): > t0 = time.time() > for _ in range(repeat): > list(gen(depth)) > t1 = time.time() > print('{}({}) * {}: total {:.3f}s'.format( > gen.__name__, depth, repeat, t1-t0)) > > The result is that there is no observable performance difference. Minimum > timing of 3 runs > > :: > > abinary(19) * 30: total 12.985s > binary(19) * 30: total 12.953s > > Note that depth of 19 means 1,048,575 calls. > > > Reference Implementation > ======================== > > The reference implementation can be found here: [3]_. > > List of high-level changes and new protocols > -------------------------------------------- > > 1. New syntax for defining coroutines: ``async def`` and new ``await`` > keyword. > > 2. New ``__await__`` method for Future-like objects. > > 3. New syntax for asynchronous context managers: ``async with``. And > associated protocol with ``__aenter__`` and ``__aexit__`` methods. > > 4. New syntax for asynchronous iteration: ``async for``. And associated > protocol with ``__aiter__``, ``__aexit__`` and new built-in exception > ``StopAsyncIteration``. > > 5. New AST nodes: ``AsyncFor``, ``AsyncWith``, ``Await``; ``FunctionDef`` > AST > node got a new argument ``is_async``. > > 6. New functions: ``sys.set_async_wrapper(callback)`` and > ``types.async_def(gen)``. > > 7. New ``CO_ASYNC`` bit flag for code objects. > > While the list of changes and new things is not short, it is important to > understand, that most users will not use these features directly. It is > intended to be used in frameworks and libraries to provide users with > convenient to use and unambiguous APIs with ``async def``, ``await``, > ``async > for`` and ``async with`` syntax. > > > Working example > --------------- > > All concepts proposed in this PEP are implemented [3]_ and can be tested. > > :: > > import asyncio > > > async def echo_server(): > print('Serving on localhost:8000') > await asyncio.start_server(handle_connection, 'localhost', 8000) > > > async def handle_connection(reader, writer): > print('New connection...') > > while True: > data = await reader.read(8192) > > if not data: > break > > print('Sending {:.10}... back'.format(repr(data))) > writer.write(data) > > > loop = asyncio.get_event_loop() > loop.run_until_complete(echo_server()) > try: > loop.run_forever() > finally: > loop.close() > > > References > ========== > > .. [1] > https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine > > .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions > > .. [3] https://github.com/1st1/cpython/tree/await > > .. [4] https://hg.python.org/benchmarks > > .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx > > .. [6] http://docs.hhvm.com/manual/en/hack.async.php > > .. [7] https://www.dartlang.org/articles/await-async/ > > .. [8] http://docs.scala-lang.org/sips/pending/async.html > > .. [9] > https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental > > .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf > (PDF) > > > Acknowledgments > =============== > > I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew > Svetlov, > and ?ukasz Langa for their initial feedback. > > > Copyright > ========= > > This document has been placed in the public domain. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmludo at gmail.com Sat Apr 18 00:38:21 2015 From: gmludo at gmail.com (Ludovic Gasc) Date: Fri, 17 Apr 2015 15:38:21 -0700 (PDT) Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: <4dd31123-ef09-4ee4-8773-4d318b1a3c88@googlegroups.com> As a lambda end-user of AsyncIO, it seems promising. As I said to Andrew Svetlov during PyCON 2015, I really prefer this new syntax. For a new comer, it will be more easier to handle because you don't need to explain the first time what is a decorator, a yield, and a yield from in the same time that Futures, Coroutines and Tasks. It's a (small) distraction about Python internal plumber about AsyncIO implementation, instead to concentrate on async business logic implementation. Nevertheless, the only small fear I have is a potential "schism" in AsyncIO community like we have with Python 2/3. For end-products source code, no problems: For new projects you can use the new syntax, or migrate easily old projects, thanks to compatibility plan. But for open source libraries, you must keep old syntax is you want Python 3.3+ compatibility. I know it's a small change compare to Python 3 changes, and AsyncIO community is enough small for now and almost AsyncIO users are early adopters for now: It means it will be more fluid for the shift. Because async keyword integration will change the end-user source code, I vote for an integration in Python 3.5 (not 3.6): it will reduce the waiting time for the end-users to use that. In an ideal world, it should be useful if we could also use that with Python 3.3 and 3.4 to help migration, because mainstream Linux distributions for production (Debian Jessie, Ubuntu 14.04 and CentOS/RHEL 7) have packages for Python 3.4, not Python 3.5, and it will change only for the new versions of theses distributions. Nevertheless, I understand the technical and political reasons that it's impossible to do that: It's an internal change of CPython, can't be added via a third library. Hopefully, it's really easy to compile CPython, an AsyncIO early adopter can't be afraid to do that ;-) -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ On Friday, April 17, 2015 at 8:59:29 PM UTC+2, Yury Selivanov wrote: > > Hello python-ideas, > > Here's my proposal to add async/await in Python. > > I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, > and I hope that this PEP can be another big step. Who knows, maybe it > will be one of the reasons to drive people's interest towards Python 3. > > > PEP: XXX > Title: Coroutines with async and await syntax > Version: $Revision$ > Last-Modified: $Date$ > Author: Yury Selivanov > > Discussions-To: Python-Dev > > Python-Version: 3.5 > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 09-Apr-2015 > Post-History: > Resolution: > > > Abstract > ======== > > This PEP introduces new syntax for coroutines, asynchronous ``with`` > statements and ``for`` loops. The main motivation behind this proposal > is to > streamline writing and maintaining asynchronous code, as well as to > simplify > previously hard to implement code patterns. > > > Rationale and Goals > =================== > > Current Python supports implementing coroutines via generators (PEP 342), > further enhanced by the ``yield from`` syntax introduced in PEP 380. > This approach has a number of shortcomings: > > * it is easy to confuse coroutines with regular generators, since they > share > the same syntax; async libraries often attempt to alleviate this by > using > decorators (e.g. ``@asyncio.coroutine`` [1]_); > > * it is not possible to natively define a coroutine which has no ``yield`` > or ``yield from`` statements, again requiring the use of decorators to > fix potential refactoring issues; > > * support for asynchronous calls is limited to expressions where > ``yield`` is > allowed syntactically, limiting the usefulness of syntactic features, > such > as ``with`` and ``for`` statements. > > This proposal makes coroutines a native Python language feature, and > clearly > separates them from generators. This removes generator/coroutine > ambiguity, > and makes it possible to reliably define coroutines without reliance on a > specific library. This also enables linters and IDEs to improve static > code > analysis and refactoring. > > Native coroutines and the associated new syntax features make it possible > to define context manager and iteration protocols in asynchronous terms. > As shown later in this proposal, the new ``async with`` statement lets > Python > programs perform asynchronous calls when entering and exiting a runtime > context, and the new ``async for`` statement makes it possible to perform > asynchronous calls in iterators. > > > Specification > ============= > > This proposal introduces new syntax and semantics to enhance coroutine > support > in Python, it does not change the internal implementation of coroutines, > which > are still based on generators. > > It is strongly suggested that the reader understands how coroutines are > implemented in Python (PEP 342 and PEP 380). It is also recommended to > read > PEP 3156 (asyncio framework). > > From this point in this document we use the word *coroutine* to refer to > functions declared using the new syntax. *generator-based coroutine* is > used > where necessary to refer to coroutines that are based on generator syntax. > > > New Coroutine Declaration Syntax > -------------------------------- > > The following new syntax is used to declare a coroutine:: > > async def read_data(db): > pass > > Key properties of coroutines: > > * Coroutines are always generators, even if they do not contain ``await`` > expressions. > > * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions > in > an ``async`` function. > > * Internally, a new code object flag - ``CO_ASYNC`` - is introduced to > enable > runtime detection of coroutines (and migrating existing code). > All coroutines have both ``CO_ASYNC`` and ``CO_GENERATOR`` flags set. > > * Regular generators, when called, return a *generator object*; similarly, > coroutines return a *coroutine object*. > > * ``StopIteration`` exceptions are not propagated out of coroutines, and > are > replaced with a ``RuntimeError``. For regular generators such behavior > requires a future import (see PEP 479). > > > types.async_def() > ----------------- > > A new function ``async_def(gen)`` is added to the ``types`` module. It > applies ``CO_ASYNC`` flag to the passed generator's code object, so that it > returns a *coroutine object* when called. > > This feature enables an easy upgrade path for existing libraries. > > > Await Expression > ---------------- > > The following new ``await`` expression is used to obtain a result of > coroutine > execution:: > > async def read_data(db): > data = await db.fetch('SELECT ...') > ... > > ``await``, similarly to ``yield from``, suspends execution of ``read_data`` > coroutine until ``db.fetch`` *awaitable* completes and returns the result > data. > > It uses the ``yield from`` implementation with an extra step of > validating its > argument. ``await`` only accepts an *awaitable*, which can be one of: > > * A *coroutine object* returned from a coroutine or a generator > decorated with > ``types.async_def()``. > > * An object with an ``__await__`` method returning an iterator. > > Any ``yield from`` chain of calls ends with a ``yield``. This is a > fundamental mechanism of how *Futures* are implemented. Since, > internally, > coroutines are a special kind of generators, every ``await`` is > suspended by > a ``yield`` somewhere down the chain of ``await`` calls (please refer > to PEP > 3156 for a detailed explanation.) > > To enable this behavior for coroutines, a new magic method called > ``__await__`` is added. In asyncio, for instance, to enable Future > objects > in ``await`` statements, the only change is to add ``__await__ = > __iter__`` > line to ``asyncio.Future`` class. > > Objects with ``__await__`` method are called *Future-like* objects in > the > rest of this PEP. > > Also, please note that ``__aiter__`` method (see its definition > below) cannot > be used for this purpose. It is a different protocol, and would be like > using ``__iter__`` instead of ``__call__`` for regular callables. > > It is a ``SyntaxError`` to use ``await`` outside of a coroutine. > > > Asynchronous Context Managers and "async with" > ---------------------------------------------- > > An *asynchronous context manager* is a context manager that is able to > suspend > execution in its *enter* and *exit* methods. > > To make this possible, a new protocol for asynchronous context managers is > proposed. Two new magic methods are added: ``__aenter__`` and > ``__aexit__``. > Both must return an *awaitable*. > > An example of an asynchronous context manager:: > > class AsyncContextManager: > async def __aenter__(self): > await log('entering context') > > async def __aexit__(self, exc_type, exc, tb): > await log('exiting context') > > > New Syntax > '''''''''' > > A new statement for asynchronous context managers is proposed:: > > async with EXPR as VAR: > BLOCK > > > which is semantically equivalent to:: > > mgr = (EXPR) > aexit = type(mgr).__aexit__ > aenter = type(mgr).__aenter__(mgr) > exc = True > > try: > try: > VAR = await aenter > BLOCK > except: > exc = False > exit_res = await aexit(mgr, *sys.exc_info()) > if not exit_res: > raise > > finally: > if exc: > await aexit(mgr, None, None, None) > > > As with regular ``with`` statements, it is possible to specify multiple > context > managers in a single ``async with`` statement. > > It is an error to pass a regular context manager without ``__aenter__`` and > ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` to use > ``async with`` outside of a coroutine. > > > Example > ''''''' > > With asynchronous context managers it is easy to implement proper database > transaction managers for coroutines:: > > async def commit(session, data): > ... > > async with session.transaction(): > ... > await session.update(data) > ... > > Code that needs locking also looks lighter:: > > async with lock: > ... > > instead of:: > > with (yield from lock): > ... > > > Asynchronous Iterators and "async for" > -------------------------------------- > > An *asynchronous iterable* is able to call asynchronous code in its *iter* > implementation, and *asynchronous iterator* can call asynchronous code > in its > *next* method. To support asynchronous iteration: > > 1. An object must implement an ``__aiter__`` method returning an > *awaitable* > resulting in an *asynchronous iterator object*. > > 2. An *asynchronous iterator object* must implement an ``__anext__`` method > returning an *awaitable*. > > 3. To stop iteration```__anext__`` must raise a ``StopAsyncIteration`` > exception. > > An example of asynchronous iterable:: > > class AsyncIterable: > async def __aiter__(self): > return self > > async def __anext__(self): > data = await self.fetch_data() > if data: > return data > else: > raise StopAsyncIteration > > async def fetch_data(self): > ... > > > New Syntax > '''''''''' > > A new statement for iterating through asynchronous iterators is proposed:: > > async for TARGET in ITER: > BLOCK > else: > BLOCK2 > > which is semantically equivalent to:: > > iter = (ITER) > iter = await type(iter).__aiter__(iter) > running = True > while running: > try: > TARGET = await type(iter).__anext__(iter) > except StopAsyncIteration: > running = False > else: > BLOCK > else: > BLOCK2 > > > It is an error to pass a regular iterable without ``__aiter__`` method to > ``async for``. It is a ``SyntaxError`` to use ``async for`` outside of a > coroutine. > > As for with regular ``for`` statement, ``async for`` has an optional > ``else`` > clause. > > > Example 1 > ''''''''' > > With asynchronous iteration protocol it is possible to asynchronously > buffer > data during iteration:: > > async for data in cursor: > ... > > Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows > of data from a database after every ``N`` iterations. > > The following code illustrates new asynchronous iteration protocol:: > > class Cursor: > def __init__(self): > self.buffer = collections.deque() > > def _prefetch(self): > ... > > async def __aiter__(self): > return self > > async def __anext__(self): > if not self.buffer: > self.buffer = await self._prefetch() > if not self.buffer: > raise StopAsyncIteration > return self.buffer.popleft() > > then the ``Cursor`` class can be used as follows:: > > async for row in Cursor(): > print(row) > > which would be equivalent to the following code:: > > i = await Cursor().__aiter__() > while True: > try: > row = await i.__anext__() > except StopAsyncIteration: > break > else: > print(row) > > > Example 2 > ''''''''' > > The following is a utility class that transforms a regular iterable to an > asynchronous one. While this is not a very useful thing to do, the code > illustrates the relationship between regular and asynchronous iterators. > > :: > > class AsyncIteratorWrapper: > def __init__(self, obj): > self._it = iter(obj) > > async def __aiter__(self): > return self > > async def __anext__(self): > try: > value = next(self._it) > except StopIteration: > raise StopAsyncIteration > return value > > data = "abc" > it = AsyncIteratorWrapper("abc") > async for item in it: > print(it) > > > Why StopAsyncIteration? > ''''''''''''''''''''''' > > Coroutines are still based on generators internally. So, before PEP > 479, there > was no fundamental difference between > > :: > > def g1(): > yield from fut > return 'spam' > > and > > :: > > def g2(): > yield from fut > raise StopIteration('spam') > > And since PEP 479 is accepted and enabled by default for coroutines, the > following example will have its ``StopIteration`` wrapped into a > ``RuntimeError`` > > :: > > async def a1(): > await fut > raise StopIteration('spam') > > The only way to tell the outside code that the iteration has ended is to > raise > something other than ``StopIteration``. Therefore, a new built-in > exception > class ``StopAsyncIteration`` was added. > > Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions > raised > in coroutines are wrapped in ``RuntimeError``. > > > Debugging Features > ------------------ > > One of the most frequent mistakes that people make when using generators as > coroutines is forgetting to use ``yield from``:: > > @asyncio.coroutine > def useful(): > asyncio.sleep(1) # this will do noting without 'yield from' > > For debugging this kind of mistakes there is a special debug mode in > asyncio, > in which ``@coroutine`` decorator wraps all functions with a special object > with a destructor logging a warning. Whenever a wrapped generator gets > garbage > collected, a detailed logging message is generated with information > about where > exactly the decorator function was defined, stack trace of where it was > collected, etc. Wrapper object also provides a convenient ``__repr__`` > function with detailed information about the generator. > > The only problem is how to enable these debug capabilities. Since debug > facilities should be a no-op in production mode, ``@coroutine`` > decorator makes > the decision of whether to wrap or not to wrap based on an OS environment > variable ``PYTHONASYNCIODEBUG``. This way it is possible to run asyncio > programs with asyncio's own functions instrumented. > ``EventLoop.set_debug``, a > different debug facility, has no impact on ``@coroutine`` decorator's > behavior. > > With this proposal, coroutines is a native, distinct from generators, > concept. A new method ``set_async_wrapper`` is added to the ``sys`` > module, > with which frameworks can provide advanced debugging facilities. > > It is also important to make coroutines as fast and efficient as possible, > therefore there are no debug features enabled by default. > > Example:: > > async def debug_me(): > await asyncio.sleep(1) > > def async_debug_wrap(generator): > return asyncio.AsyncDebugWrapper(generator) > > sys.set_async_wrapper(async_debug_wrap) > > debug_me() # <- this line will likely GC the coroutine object and > # trigger AsyncDebugWrapper's code. > > assert isinstance(debug_me(), AsyncDebugWrapper) > > sys.set_async_wrapper(None) # <- this unsets any previously set > wrapper > assert not isinstance(debug_me(), AsyncDebugWrapper) > > If ``sys.set_async_wrapper()`` is called twice, the new wrapper replaces > the > previous wrapper. ``sys.set_async_wrapper(None)`` unsets the wrapper. > > > Glossary > ======== > > :Coroutine: > A coroutine function, or just "coroutine", is declared with ``async > def``. > It uses ``await`` and ``return value``; see `New Coroutine Declaration > Syntax`_ for details. > > :Coroutine object: > Returned from a coroutine function. See `Await Expression`_ for > details. > > :Future-like object: > An object with an ``__await__`` method. It is consumed by > ``await`` in a > coroutine. A coroutine waiting for a Future-like object is > suspended until > the Future-like object's ``__await__`` completes. ``await`` > returns the > result of the Future-like object. See `Await Expression`_ for > details. > > :Awaitable: > A *future-like* object or a *coroutine object*. See `Await > Expression`_ > for details. > > :Generator-based coroutine: > Coroutines based in generator syntax. Most common example is > ``@asyncio.coroutine``. > > :Asynchronous context manager: > An asynchronous context manager has ``__aenter__`` and ``__aexit__`` > methods > and can be used with ``async with``. See > `Asynchronous Context Managers and "async with"`_ for details. > > :Asynchronous iterable: > An object with an ``__aiter__`` method, which must return an > *asynchronous > iterator* object. Can be used with ``async for``. See > `Asynchronous Iterators and "async for"`_ for details. > > :Asynchronous iterator: > An asynchronous iterator has an ``__anext__`` method.See > `Asynchronous Iterators and "async for"`_ for details. > > > List of functions and methods > ============================= > > ================= ======================================= ================= > Method Can contain Can't contain > ================= ======================================= ================= > async def func await, return value yield, yield > from > async def __a*__ await, return value yield, yield > from > def __a*__ return Future-like await > def __await__ yield, yield from, return iterable await > generator yield, yield from, return value await > ================= ======================================= ================= > > Where: > > * ""async def func": coroutine; > > * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, > ``__aexit__`` defined with the ``async`` keyword; > > * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, ``__aexit__`` > defined without the ``async`` keyword, must return an *awaitable*; > > * "def __await__": ``__await__`` method to implement *Future-like* objects; > > * generator: a "regular" generator, function defined with ``def`` and which > contains a least one ``yield`` or ``yield from`` expression. > > *Future-like* is an object with an ``__await__`` method, see > `Await Expression`_ section for details. > > > Transition Plan > =============== > > To avoid backwards compatibility issues with ``async`` and ``await`` > keywords, > it was decided to modify ``tokenizer.c`` in such a way, that it: > > * recognizes ``async def`` name tokens combination (start of a coroutine); > > * keeps track of regular functions and coroutines; > > * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with > ``AWAIT`` when in the process of yielding tokens for coroutines. > > This approach allows for seamless combination of new syntax features (all > of > them available only in ``async`` functions) with any existing code. > > An example of having "async def" and "async" attribute in one piece of > code:: > > class Spam: > async = 42 > > async def ham(): > print(getattr(Spam, 'async')) > > # The coroutine can be executed and will print '42' > > > Backwards Compatibility > ----------------------- > > The only backwards incompatible change is an extra argument ``is_async`` to > ``FunctionDef`` AST node. But since it is a documented fact that the > structure > of AST nodes is an implementation detail and subject to change, this > should not > be considered a serious issue. > > > Grammar Updates > --------------- > > Grammar changes are also fairly minimal:: > > await_expr: AWAIT test > await_stmt: await_expr > > decorated: decorators (classdef | funcdef | async_funcdef) > async_funcdef: ASYNC funcdef > > async_stmt: ASYNC (funcdef | with_stmt) # will add for_stmt later > > compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt > | funcdef | classdef | decorated | async_stmt) > > atom: ('(' [yield_expr|await_expr|testlist_comp] ')' | > '[' [testlist_comp] ']' | > '{' [dictorsetmaker] '}' | > NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False?) > > expr_stmt: testlist_star_expr (augassign > (yield_expr|await_expr|testlist) | > ('=' (yield_expr|await_expr|testlist_star_expr))*) > > > Transition Period Shortcomings > ------------------------------ > > There is just one. > > Until ``async`` and ``await`` are not proper keywords, it is not > possible (or > at least very hard) to fix ``tokenizer.c`` to recognize them on the **same > line** with ``def`` keyword:: > > # async and await will always be parsed as variables > > async def outer(): # 1 > def nested(a=(await fut)): > pass > > async def foo(): return (await fut) # 2 > > Since ``await`` and ``async`` in such cases are parsed as ``NAME`` tokens, > a > ``SyntaxError`` will be raised. > > To workaround these issues, the above examples can be easily rewritten to a > more readable form:: > > async def outer(): # 1 > a_default = await fut > def nested(a=a_default): > pass > > async def foo(): # 2 > return (await fut) > > This limitation will go away as soon as ``async`` and ``await`` ate proper > keywords. Or if it's decided to use a future import for this PEP. > > > Deprecation Plans > ----------------- > > ``async`` and ``await`` names will be softly deprecated in CPython 3.5 > and 3.6. > In 3.7 we will transform them to proper keywords. Making ``async`` and > ``await`` proper keywords before 3.7 might make it harder for people to > port > their code to Python 3. > > > asyncio > ------- > > ``asyncio`` module was adapted and tested to work with coroutines and new > statements. Backwards compatibility is 100% preserved. > > The required changes are mainly: > > 1. Modify ``@asyncio.coroutine`` decorator to use new ``types.async_def()`` > function. > > 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. > > 3. Add ``ensure_task()`` as an alias for ``async()`` function. Deprecate > ``async()`` function. > > > Design Considerations > ===================== > > No implicit wrapping in Futures > ------------------------------- > > There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A key > difference is that JavaScript "async functions" always return a Promise. > While > this approach has some advantages, it also implies that a new Promise > object is > created on each "async function" invocation. > > We could implement a similar functionality in Python, by wrapping all > coroutines in a Future object, but this has the following disadvantages: > > 1. Performance. A new Future object would be instantiated on each > coroutine > call. Moreover, this makes implementation of ``await`` expressions > slower > (disabling optimizations of ``yield from``). > > 2. A new built-in ``Future`` object would need to be added. > > 3. Coming up with a generic ``Future`` interface that is usable for any use > case in any framework is a very hard to solve problem. > > 4. It is not a feature that is used frequently, when most of the code is > coroutines. > > > Why "async" and "await" keywords > -------------------------------- > > async/await is not a new concept in programming languages: > > * C# has it since long time ago [5]_; > > * proposal to add async/await in ECMAScript 7 [2]_; > see also Traceur project [9]_; > > * Facebook's Hack/HHVM [6]_; > > * Google's Dart language [7]_; > > * Scala [8]_; > > * proposal to add async/await to C++ [10]_; > > * and many other less popular languages. > > This is a huge benefit, as some users already have experience with > async/await, > and because it makes working with many languages in one project easier > (Python > with ECMAScript 7 for instance). > > > Why "__aiter__" is a coroutine > ------------------------------ > > In principle, ``__aiter__`` could be a regular function. There are several > good reasons to make it a coroutine: > > * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` > methods are > coroutines, users would often make a mistake defining it as ``async`` > anyways; > > * there might be a need to run some asynchronous operations in > ``__aiter__``, > for instance to prepare DB queries or do some file operation. > > > Importance of "async" keyword > ----------------------------- > > While it is possible to just implement ``await`` expression and treat all > functions with at least one ``await`` as coroutines, this approach makes > APIs design, code refactoring and its long time support harder. > > Let's pretend that Python only has ``await`` keyword:: > > def useful(): > ... > await log(...) > ... > > def important(): > await useful() > > If ``useful()`` function is refactored and someone removes all ``await`` > expressions from it, it would become a regular python function, and all > code > that depends on it, including ``important()`` would be broken. To > mitigate this > issue a decorator similar to ``@asyncio.coroutine`` has to be introduced. > > > Why "async def" > --------------- > > For some people bare ``async name(): pass`` syntax might look more > appealing > than ``async def name(): pass``. It is certainly easier to type. But on > the > other hand, it breaks the symmetry between ``async def``, ``async with`` > and > ``async for``, where ``async`` is a modifier, stating that the statement is > asynchronous. It is also more consistent with the existing grammar. > > > Why not a __future__ import > --------------------------- > > ``__future__`` imports are inconvenient and easy to forget to add. Also, > they > are enabled for the whole source file. Consider that there is a big > project > with a popular module named "async.py". With future imports it is > required to > either import it using ``__import__()`` or ``importlib.import_module()`` > calls, > or to rename the module. The proposed approach makes it possible to > continue > using old code and modules without a hassle, while coming up with a > migration > plan for future python versions. > > > Why magic methods start with "a" > -------------------------------- > > New asynchronous magic methods ``__aiter__``, ``__anext__``, > ``__aenter__``, > and ``__aexit__`` all start with the same prefix "a". An alternative > proposal > is to use "async" prefix, so that ``__aiter__`` becomes ``__async_iter__``. > However, to align new magic methods with the existing ones, such as > ``__radd__`` and ``__iadd__`` it was decided to use a shorter version. > > > Why not reuse existing magic names > ---------------------------------- > > An alternative idea about new asynchronous iterators and context > managers was > to reuse existing magic methods, by adding an ``async`` keyword to their > declarations:: > > class CM: > async def __enter__(self): # instead of __aenter__ > ... > > This approach has the following downsides: > > * it would not be possible to create an object that works in both > ``with`` and > ``async with`` statements; > > * it would look confusing and would require some implicit magic behind the > scenes in the interpreter; > > * one of the main points of this proposal is to make coroutines as simple > and foolproof as possible. > > > Comprehensions > -------------- > > For the sake of restricting the broadness of this PEP there is no new > syntax > for asynchronous comprehensions. This should be considered in a > separate PEP, > if there is a strong demand for this feature. > > > Performance > =========== > > Overall Impact > -------------- > > This proposal introduces no observable performance impact. Here is an > output > of python's official set of benchmarks [4]_: > > :: > > python perf.py -r -b default ../cpython/python.exe > ../cpython-aw/python.exe > > [skipped] > > Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: > Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 > x86_64 i386 > > Total CPU cores: 8 > > ### etree_iterparse ### > Min: 0.365359 -> 0.349168: 1.05x faster > Avg: 0.396924 -> 0.379735: 1.05x faster > Significant (t=9.71) > Stddev: 0.01225 -> 0.01277: 1.0423x larger > > The following not significant results are hidden, use -v to show them: > django_v2, 2to3, etree_generate, etree_parse, etree_process, > fastpickle, > fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. > > > Tokenizer modifications > ----------------------- > > There is no observable slowdown of parsing python files with the modified > tokenizer: parsing of one 12Mb file (``Lib/test/test_binop.py`` repeated > 1000 > times) takes the same amount of time. > > > async/await > ----------- > > The following micro-benchmark was used to determine performance difference > between "async" functions and generators:: > > import sys > import time > > def binary(n): > if n <= 0: > return 1 > l = yield from binary(n - 1) > r = yield from binary(n - 1) > return l + 1 + r > > async def abinary(n): > if n <= 0: > return 1 > l = await abinary(n - 1) > r = await abinary(n - 1) > return l + 1 + r > > def timeit(gen, depth, repeat): > t0 = time.time() > for _ in range(repeat): > list(gen(depth)) > t1 = time.time() > print('{}({}) * {}: total {:.3f}s'.format( > gen.__name__, depth, repeat, t1-t0)) > > The result is that there is no observable performance difference. Minimum > timing of 3 runs > > :: > > abinary(19) * 30: total 12.985s > binary(19) * 30: total 12.953s > > Note that depth of 19 means 1,048,575 calls. > > > Reference Implementation > ======================== > > The reference implementation can be found here: [3]_. > > List of high-level changes and new protocols > -------------------------------------------- > > 1. New syntax for defining coroutines: ``async def`` and new ``await`` > keyword. > > 2. New ``__await__`` method for Future-like objects. > > 3. New syntax for asynchronous context managers: ``async with``. And > associated protocol with ``__aenter__`` and ``__aexit__`` methods. > > 4. New syntax for asynchronous iteration: ``async for``. And associated > protocol with ``__aiter__``, ``__aexit__`` and new built-in exception > ``StopAsyncIteration``. > > 5. New AST nodes: ``AsyncFor``, ``AsyncWith``, ``Await``; > ``FunctionDef`` AST > node got a new argument ``is_async``. > > 6. New functions: ``sys.set_async_wrapper(callback)`` and > ``types.async_def(gen)``. > > 7. New ``CO_ASYNC`` bit flag for code objects. > > While the list of changes and new things is not short, it is important to > understand, that most users will not use these features directly. It is > intended to be used in frameworks and libraries to provide users with > convenient to use and unambiguous APIs with ``async def``, ``await``, > ``async > for`` and ``async with`` syntax. > > > Working example > --------------- > > All concepts proposed in this PEP are implemented [3]_ and can be tested. > > :: > > import asyncio > > > async def echo_server(): > print('Serving on localhost:8000') > await asyncio.start_server(handle_connection, 'localhost', 8000) > > > async def handle_connection(reader, writer): > print('New connection...') > > while True: > data = await reader.read(8192) > > if not data: > break > > print('Sending {:.10}... back'.format(repr(data))) > writer.write(data) > > > loop = asyncio.get_event_loop() > loop.run_until_complete(echo_server()) > try: > loop.run_forever() > finally: > loop.close() > > > References > ========== > > .. [1] > https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine > > .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions > > .. [3] https://github.com/1st1/cpython/tree/await > > .. [4] https://hg.python.org/benchmarks > > .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx > > .. [6] http://docs.hhvm.com/manual/en/hack.async.php > > .. [7] https://www.dartlang.org/articles/await-async/ > > .. [8] http://docs.scala-lang.org/sips/pending/async.html > > .. [9] > > https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental > > .. [10] > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF) > > > Acknowledgments > =============== > > I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew > Svetlov, > and ?ukasz Langa for their initial feedback. > > > Copyright > ========= > > This document has been placed in the public domain. > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Sat Apr 18 19:02:32 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 18 Apr 2015 13:02:32 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <5531A390.8060405@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> Message-ID: <55328E28.4010608@gmail.com> Hi Greg, On 2015-04-17 8:21 PM, Greg Ewing wrote: > Yury Selivanov wrote: >> >> Here's my proposal to add async/await in Python. > > You've essentially reinvented PEP 3152 - Cofunctions. > > https://www.python.org/dev/peps/pep-3152/ > > Here's a summary of the relationships between them: > > PEP 3152 Nearest equivalent in Yury's PEP > -------- -------------------------------- > > codef f(args): async def f(args): > > cocall f(args) await f(args) > > __cocall__ __await__ > > costart() async_def() Let me list the key differences here: 1. Keywords, that's obvious. To be honest, I don't like cocall, codef, cofor, and cowith. That's my personal, highly subjective, opinion, though. 2. I don't like the special relationship of 'cocall' token and parens in the grammar. What's going on here: 'cocall foo()()'? And what's 'cocall foo().bar()'? Will it raise a SyntaxError in the former case? And in the latter -- are you cocalling 'foo().bar' or 'foo()'? These are rhetorical questions, but in real life people will ask them. 3. 'costart' is not an equivalent of 'async_def'. 'async_def' allows you to wrap existing generator-based coroutines into the proposed 'async def' ones. By using 'async_def' in asyncio.coroutine decorator we can seamlessly enable all existing asyncio code to be used in 'await' expression. I don't see how you can do this with costart(). Probably you need some kind of a wrapper object, like that: class CoCoro: def __cocall__(self, *args, **kwargs): return (yield from self.__wrapped__) and apply it to all asyncio coroutines. That's some extra tax right here. While 'async_def' is simply flipping a CO_ASYNC bit. 4. Lack of 'async with' and 'async for', obvious again. > > There is currently no equivalent of "async for" and > "async with" in PEP 3152, but they could easily be added. > I would probably spell them "cofor" and "cowith". > > As the author of PEP 3152 I'm obviously biased, but I > think my spellings are more elegant and less disruptive > to reading of the code. > > PEP 3152 is currently marked as deferred. Maybe it's > time to revive it? If Yury's pep is to be considered, > we ought to discuss the relative merits of the two. > Thank you, Yury From yselivanov.ml at gmail.com Sat Apr 18 19:04:54 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 18 Apr 2015 13:04:54 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: <55328EB6.9060906@gmail.com> Hi Todd, On 2015-04-18 6:41 AM, Todd wrote: > On Apr 17, 2015 8:59 PM, "Yury Selivanov" wrote: >> >> >> async with EXPR as VAR: >> BLOCK >> >> >> >> >> async for TARGET in ITER: >> BLOCK >> else: >> BLOCK2 >> > Is this really something that should restricted to async? I could see > similar syntax being useful for multithreading, multiprocessing, MPI, etc. > What about a more general language feature, where any class implementing > certain magic methods could be used in this way? Coroutines in PEP 492 is a generic concept, completely framework agnostic. Any class can implement the methods, coroutines can be used in any way you like. asyncio users will definitely take full advantage of new syntax, but nothing prevents others to create new libraries and invent new use cases. Thanks, Yury From yselivanov.ml at gmail.com Sat Apr 18 19:05:58 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 18 Apr 2015 13:05:58 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <5531869C.8090804@gmail.com> <55318FA5.6010403@gmail.com> Message-ID: <55328EF6.5070805@gmail.com> Hi Brett, On 2015-04-18 9:23 AM, Brett Cannon wrote: > +1 from me on the PEP! Very thorough and well argued. Thanks a lot! > > On Fri, Apr 17, 2015 at 6:57 PM Yury Selivanov > wrote: > >> Chris, >> On 2015-04-17 6:46 PM, Chris Angelico wrote: >>> On Sat, Apr 18, 2015 at 8:18 AM, Yury Selivanov >> wrote: >>>> On 2015-04-17 6:00 PM, Chris Angelico wrote: >>>>> On Sat, Apr 18, 2015 at 4:58 AM, Yury Selivanov < >> yselivanov.ml at gmail.com> >>>>> wrote: >> [SNIP] >> >>>> Let's see what python-ideas thinks about it. I'm fine if everybody >>>> wants __future__ imports. I will only have to rollback my changes >>>> in tokenizer.c and change few tokens in Grammar to make the >>>> reference implementation work. >>> In case it wasn't clear from my previous post, I'm +1 on using a >>> __future__ import. Victor's idea of an optional directive seems >>> interesting, but I'm not sure how useful it'd be in reality; does >>> complicating the rules offer more benefit than simply having a keyword >>> governed by __future__? >> I'm OK with __future__ import. And I'll be extremely happy >> if that's the only thing we'll be all discussing here ;-) >> >> Let's see how the discussion goes, and if everybody on the >> list wants __future__ and Guido approves, I'll update >> the PEP and ref implementation! >> > +1 for the __future__ statement. I'm actually leaning towards it too... Yury From steve at pearwood.info Sat Apr 18 19:11:45 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 19 Apr 2015 03:11:45 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: <553157F2.8060408@gmail.com> References: <553157F2.8060408@gmail.com> Message-ID: <20150418171140.GY5663@ando.pearwood.info> Nicely written and very complete! I only have a couple of little niggles, which some might consider bike-shedding. See below. On Fri, Apr 17, 2015 at 02:58:58PM -0400, Yury Selivanov wrote: [...] > From this point in this document we use the word *coroutine* to refer to > functions declared using the new syntax. *generator-based coroutine* is > used where necessary to refer to coroutines that are based on > generator syntax. It concerns me that we will have two kinds of coroutines. At the moment, we have two kinds of generator functions: those that are used as generators, and those that are used as coroutines, depending on whether they contain "yield expr" or "x = (yield expr)". That distinction won't go away, but now we will have even more ambiguity of language: there will be generators which are functions: def gen(): yield 1 generators which are not functions but the result of calling a generator function: g = gen() coroutines which are generators: def gen(): x = (yield 1) y = (yield x+1) cr = gen() and coroutines which are not generators: async def g(): await spam() Except that they will be implemented internally as generators. There will be generators that use yield and generators which give a syntax error with yield. My head hurts :-) I think it will be hard to unambiguously talk about these things without confusion, especially for people with limited asyncronous experience. I don't have an answer to this, but I know that even the generator versus generator-function ambiguity (both get called "generator") sometimes leads to confusion. (E.g. below you talk about passing a generator to types.async_def but it isn't clear to me whether you mean the generator or the generator function.) There's been at least one thread on python-list in the last few months about that. Is it possible to use another name to distinguish generator-based coroutines from async-based coroutines? It's a pity we can't steal the name "goroutine" from Go. I've always liked that name :-) but as I understand it, goroutines completely unrelated to coroutines. Hmmm... maybe we could use pyroutines? *wink* If the name must stay as-is, it would help to have an unambigous and clear summary of the various generator-or-coroutine uses and how they differ. > types.async_def() > ----------------- > > A new function ``async_def(gen)`` is added to the ``types`` module. It > applies ``CO_ASYNC`` flag to the passed generator's code object, so that it > returns a *coroutine object* when called. > > This feature enables an easy upgrade path for existing libraries. When you say "the passed generator", do you mean the generator function or the generator object? E.g. given: def gen(): yield 1 which do I use? async_def(gen()) or async_def(gen)? -- Steve From yselivanov.ml at gmail.com Sat Apr 18 19:28:47 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 18 Apr 2015 13:28:47 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> Message-ID: <5532944F.1060608@gmail.com> Hi Jim, On 2015-04-18 12:01 PM, Jim Baker wrote: > +1 > > Such support would make async coroutines work well on implementations like > Jython, as opposed to the current situation that there's a large body of > current async code, using the greenlet approach, that currently does not. This is so great to hear! Thank you for bringing this up! > > Note that efficient support would currently require linking a jar from > https://github.com/puniverse/quasar, but there's also the possibility that > will be in a future Java as well (it's certainly discussed at most JVM > Language Summits). > > Compare this proposal to what we have with greenlet and its use of a > function named "switch", which cannot be statically determined to be such a > thing. Without the static determination we get with keyword support, we > cannot mark code as being a coroutine for Quasar, so it must be assumed to > be so, at extra runtime overhead; also such support also would require that > *all* Python code runs as Python bytecode (with the extra overhead that > entails of running a Python bytecode VM on the JVM) since it could possibly > be asynchronous. > > The alternative is to map such async coroutines to threads, much as I did > for greenlet support with this proof-of-concept > https://github.com/jythontools/artificialturf Although one can potentially > run 1000s and possibly 10000s of threads on a JVM on say Linux, this is > certainly not the case for all OS targets. > > Best, Yury From steve at pearwood.info Sat Apr 18 19:38:57 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 19 Apr 2015 03:38:57 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: <20150418173857.GZ5663@ando.pearwood.info> On Sat, Apr 18, 2015 at 02:41:29PM +0300, Ionel Cristian M?rie? wrote: > On Sat, Apr 18, 2015 at 2:23 PM, Steven D'Aprano > wrote: > > > Are you sure this doesn't already work? It works for me in Python 3.3: > > > It doesn't really work because of the incomplete checks done in > `callable_builtin`. ?This is what I've tried: > > >>> class DynamicCallable: > > ... is_callable = True > > ... > > ... def __init__(self, target): > > ... self.target = target > > ... > > ... @property > > ... def __call__(self): > > ... if self.is_callable: > > ... return self.target > > ... else: > > ... raise AttributeError("Not really ...") > > ... > > >>> dc = DynamicCallable(print) > > >>> dc(1, 2, 3) > > 1 2 3 > > >>> callable(dc) > > True > > >>> dc.is_callable = False > > >>> callable(dc) > > True ###### This should be False :( Why do you think that it should be false? If you actually call dc, the __call__ method/function/property (whatever you want to name it) gets called, and it raises an exception just like you programmed it to. The traceback clearly shows __call__ in the stack. hasattr(type(dc), '__call__') returns true, therefore dc is callable. I think that the fact that the __call__ method ends up raising an AttributeError is irrelevant -- lots of callables raise AttributeError: def func(): return None.spam However, I think *this* is a bug in callable: py> class Callable: ... def __getattribute__(self, name): ... if name == '__call__': ... raise AttributeError # Oh the lies we tell. ... return super().__getattribute__(name) ... def __call__(self): ... return 23 ... py> x = Callable() py> callable(x) False py> x() 23 Clearly x is callable, since I just called it and it returned a value, but callable() thinks it is not. Here is another example: py> class X: pass ... py> x = X() py> x.__call__ = 42 py> callable(x) True py> x() Traceback (most recent call last): File "", line 1, in TypeError: 'X' object is not callable I think callable() is badly broken. (Or at least *was* broken in 3.3 -- I don't have 3.4 handy to try it.) -- Steve From yselivanov.ml at gmail.com Sat Apr 18 19:39:16 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 18 Apr 2015 13:39:16 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150418171140.GY5663@ando.pearwood.info> References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> Message-ID: <553296C4.3060603@gmail.com> Hi Steven, On 2015-04-18 1:11 PM, Steven D'Aprano wrote: > Nicely written and very complete! I only have a couple of little > niggles, which some might consider bike-shedding. See below. Thanks a lot! > > > On Fri, Apr 17, 2015 at 02:58:58PM -0400, Yury Selivanov wrote: > > [...] >> From this point in this document we use the word *coroutine* to refer to >> functions declared using the new syntax. *generator-based coroutine* is >> used where necessary to refer to coroutines that are based on >> generator syntax. > It concerns me that we will have two kinds of coroutines. > > At the moment, we have two kinds of generator functions: those that are > used as generators, and those that are used as coroutines, depending on > whether they contain "yield expr" or "x = (yield expr)". That > distinction won't go away, but now we will have even more ambiguity of > language: there will be generators which are functions: > > def gen(): yield 1 > > generators which are not functions but the result of calling a generator > function: > > g = gen() > > coroutines which are generators: > > def gen(): > x = (yield 1) > y = (yield x+1) > > cr = gen() > > and coroutines which are not generators: > > async def g(): > await spam() > > Except that they will be implemented internally as generators. There > will be generators that use yield and generators which give a syntax > error with yield. My head hurts :-) > > I think it will be hard to unambiguously talk about these things without > confusion, especially for people with limited asyncronous experience. > > I don't have an answer to this, but I know that even the generator > versus generator-function ambiguity (both get called "generator") > sometimes leads to confusion. (E.g. below you talk about passing a > generator to types.async_def but it isn't clear to me whether you mean > the generator or the generator function.) There's been at least one > thread on python-list in the last few months about that. Is it possible > to use another name to distinguish generator-based coroutines from > async-based coroutines? > > It's a pity we can't steal the name "goroutine" from Go. I've always > liked that name :-) but as I understand it, goroutines completely > unrelated to coroutines. Hmmm... maybe we could use pyroutines? *wink* > > If the name must stay as-is, it would help to have an unambigous and > clear summary of the various generator-or-coroutine uses and how they > differ. The PEP kind of addresses this, but calling current approach to do coroutines "generator-based coroutines", and the new ones as just "coroutines". The internal implementation based on generators is something that was absolutely required to cover in the PEP in a great detail, but almost everybody will forget about that fact pretty soon. The point of this PEP is to actually avoid any confusion, and to just have two separate concepts: coroutines and generators (the latter can still be used as a "coroutine-like" object, but hopefully nobody will do that at some point). > > >> types.async_def() >> ----------------- >> >> A new function ``async_def(gen)`` is added to the ``types`` module. It >> applies ``CO_ASYNC`` flag to the passed generator's code object, so that it >> returns a *coroutine object* when called. >> >> This feature enables an easy upgrade path for existing libraries. > When you say "the passed generator", do you mean the generator function or the generator object? > E.g. given: > > def gen(): > yield 1 > > which do I use? async_def(gen()) or async_def(gen)? It's generator-function. It's a very good catch. I've updated the PEP. Thanks! Yury From christian at python.org Sat Apr 18 19:43:02 2015 From: christian at python.org (Christian Heimes) Date: Sat, 18 Apr 2015 19:43:02 +0200 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: <553297A6.6040608@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 2015-04-17 23:39, Guido van Rossum wrote: > I think you've found an unintended and undocumented backdoor. I > admit I don't understand how this works in CPython. Overloaded > operators like __add__ or __call__ should be methods in the class, > and we don't look for them in the instance. But somehow defining > them with @property works (I guess because @property is in the > class). > > What's different for __call__ is that callable() exists. And this > is probably why I exorcised it Python 3.0 -- but apparently it's > back. :-( > > In the end callable() doesn't always produce a correct answer; but > maybe we can make it work in this case by first testing the class > and then the instance? Something like (untested): > > def callable(x): return hasattr(x.__class__, '__call__') and > hasattr(x, '__call__') The code behind callable() is very simple and very fast: int PyCallable_Check(PyObject *x) { if (x == NULL) return 0; return x->ob_type->tp_call != NULL; } IMHO the behavior is well in range of the documentation. It also conforms to my gut feeling and the behavior of PyPy and Jython (tested with Jython 2.7b3+ and PyPy 2.4.0). The three major Python implementation agree on callable(o) == hasattr(type(o), '__call__') for new style classes. Because PyCallable_Check() is so fast with just two pointer derefs, it may be used in several hot paths. Any modification may cause a slow down. This aspect must be thoroughly investigates before the code is changed. For all this reasons I'm -1 on the proposed change. Christian -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJVMpecAAoJEIZoUkkhLbaJyjAH/AiueFdO0wECxZkc53f10Txk Kjb1RB2SRyNIwcvOR5sXJVCP4OrazlTyDSOeCxQ50I8IBXk2vAbdKVEfjuNW4SqQ Dr6xijhA2JjAq/TfBHdMJkcGUySBPkBNTn7Dd50TvJm+PE+D4zlGXpgI7rfZXGM5 MwWrphk0/sB6bZ6WSDjdoCQ40V6CZ1uWTU2N5yd/+vtpA91Yl/FB5Xu7x3sRwt0Y A24GbJHqwgwgnQ7kFozBIbilN3dpcI+Pn5LC6KbqldlNvdp9IMCZh0dm+psnKHVq 2kClbv8f03EahScnKzVh3PblJZ2DB8AEq+PRalmi/v4m0BvWT8a073708BQLocg= =VJMz -----END PGP SIGNATURE----- From guido at python.org Sat Apr 18 22:27:01 2015 From: guido at python.org (Guido van Rossum) Date: Sat, 18 Apr 2015 13:27:01 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: On Sat, Apr 18, 2015 at 5:42 AM, Joao S. O. Bueno wrote: > On 17 April 2015 at 18:39, Guido van Rossum wrote: > > I think you've found an unintended and undocumented backdoor. I admit I > > don't understand how this works in CPython. Overloaded operators like > > __add__ or __call__ should be methods in the class, and we don't look for > > them in the instance. But somehow defining them with @property works (I > > guess because @property is in the class). > > > > What's different for __call__ is that callable() exists. And this is > > probably why I exorcised it Python 3.0 -- but apparently it's back. :-( > > And for much that I've searched I've never found out the reasoning on > that exorcism. > Since you are at it, could you describe it? > > I am glad it is back - I think it is definitely needed - even if just > works in a sort > of naive way. > Glad you asked. The reason for the exorcism was actually the kind of issues brought up in this thread -- there are a variety of edge cases where an object may in fact be called but callable() returns False, and other edge cases where callable() returns True but calling the object fails. (Not to mention that callable() says nothing about the acceptable arguments.) Not having callable() would have avoided this entire thread. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 18 23:10:58 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 19 Apr 2015 07:10:58 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sun, Apr 19, 2015 at 2:33 AM, Ionel Cristian M?rie? wrote: > On Sat, Apr 18, 2015 at 3:25 PM, Chris Angelico wrote: >> >> Is it intentional that >> property-getter functions can raise AttributeError to signal that the >> attribute does not (currently) exist? Because if that's a hack, then >> there's no reason to support it, and callable() is absolutely correct >> to say "there is a __call__ attribute on the class, ergo it's >> callable". > > > It's hard to say what's intended or not. At best is speculation, as > python-the-language does not have a specification. That's not true; there is a spec for the language, which is independent of CPython, PyPy, etc, which are the implementations of it. There are times when the spec is less than clear, which are often flagged by someone coming to python-dev saying "I'm trying to add Feature X to SomePython, and I'm not sure whether this is how it should be done or not - that's how CPython does it", and then the answer to that question becomes a language spec improvement. But in this case, the question isn't one of Python vs CPython, but one of the use of application-level code. If this is considered a hack, then it's not part of the language spec at all, but if it's deemed a feature, then (a) every Python implementation will be required to match it, and (b) other parts of the language (in this case, callable()) will probably be required to acknowledge it. ChrisA From rosuav at gmail.com Sat Apr 18 23:29:25 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 19 Apr 2015 07:29:25 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: <553296C4.3060603@gmail.com> References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> Message-ID: On Sun, Apr 19, 2015 at 3:39 AM, Yury Selivanov wrote: > The internal implementation based on generators is something > that was absolutely required to cover in the PEP in a great detail, > but almost everybody will forget about that fact pretty soon. > > The point of this PEP is to actually avoid any confusion, and > to just have two separate concepts: coroutines and generators > (the latter can still be used as a "coroutine-like" object, > but hopefully nobody will do that at some point). Which means that these are three very VERY different things, all of which happen to be implemented (under the covers) using similar mechanisms: async def corout(): await spam() def gen(): yield from [1,2,3] class iter: def __init__(self): self.state = 0 def __iter__(self): return self def __next__(self): if self.state == 3: raise StopIteration self.state += 1 return self.state Technically, a generator object is an iterator, and a coroutine is a generator, but they're really implementation details. PEP 479 is all about not expecting "raise StopIteration" to terminate gen(); in the same way, programmers should not expect "for x in corout()" to do anything meaningful, because they're completely different things. Have I understood this correctly? ChrisA From contact at ionelmc.ro Sat Apr 18 23:27:06 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sun, 19 Apr 2015 00:27:06 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sun, Apr 19, 2015 at 12:10 AM, Chris Angelico wrote: > That's not true; there is a spec for the language, which is > independent of CPython, PyPy, etc, which are the implementations of > it. > ?I'm not aware of such a document - can you point me to it? AFAIK `callable` is too old to be in a PEP.? I hope you didn't have in mind the "docs as specification", that's not really a specification, it's a "state of things". But in this case, the question isn't one of Python vs CPython, but one > of the use of application-level code. If this is considered a hack, > then it's not part of the language spec at all, but if it's deemed a > feature, then (a) every Python implementation will be required to > match it, and (b) other parts of the language (in this case, > callable()) will probably be required to acknowledge it. > I wouldn't care about other python implementations, they are years behind CPython, and this is a very small change anyway. Why would this matter for them if they still implement python-2.7-language? It doesn't really make sense, unless you are considering to fix this in 2.7 too. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Apr 18 23:34:12 2015 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 19 Apr 2015 07:34:12 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sun, Apr 19, 2015 at 7:27 AM, Ionel Cristian M?rie? wrote: > > On Sun, Apr 19, 2015 at 12:10 AM, Chris Angelico wrote: >> >> That's not true; there is a spec for the language, which is >> independent of CPython, PyPy, etc, which are the implementations of >> it. > > > I'm not aware of such a document - can you point me to it? AFAIK `callable` > is too old to be in a PEP. I hope you didn't have in mind the "docs as > specification", that's not really a specification, it's a "state of things". Not sure why the docs don't count as a spec. You're probably right that callable() doesn't have any definition outside of the main docs, but you generalized that to the entire language not having a spec, which is definitely not the case. But this thread could easily get coalesced into a simple line or two in the docs, which then _would_ be the spec. >> But in this case, the question isn't one of Python vs CPython, but one >> of the use of application-level code. If this is considered a hack, >> then it's not part of the language spec at all, but if it's deemed a >> feature, then (a) every Python implementation will be required to >> match it, and (b) other parts of the language (in this case, >> callable()) will probably be required to acknowledge it. > > I wouldn't care about other python implementations, they are years behind > CPython, and this is a very small change anyway. Why would this matter for > them if they still implement python-2.7-language? It doesn't really make > sense, unless you are considering to fix this in 2.7 too. Uhh, that's not exactly true. Maybe it's true of the other three of the Big Four (Jython, IronPython, PyPy), but there are several other Pythons which are compliant with a version 3.x spec; MicroPython and Brython come to mind. ChrisA From contact at ionelmc.ro Sat Apr 18 23:40:46 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sun, 19 Apr 2015 00:40:46 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sun, Apr 19, 2015 at 12:34 AM, Chris Angelico wrote: > > Uhh, that's not exactly true. Maybe it's true of the other three of > the Big Four (Jython, IronPython, PyPy), but there are several other > Pythons which are compliant with a version 3.x spec; MicroPython and > Brython come to mind. ?Ok, but what's reasonable here? I think you're implying here that this small change should go t?hrough a PEP process. So a question for the list, does this seemingly small change warrant a PEP? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Apr 19 01:09:48 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 19 Apr 2015 00:09:48 +0100 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On 18 April 2015 at 22:40, Ionel Cristian M?rie? wrote: > On Sun, Apr 19, 2015 at 12:34 AM, Chris Angelico wrote: >> >> >> Uhh, that's not exactly true. Maybe it's true of the other three of >> the Big Four (Jython, IronPython, PyPy), but there are several other >> Pythons which are compliant with a version 3.x spec; MicroPython and >> Brython come to mind. > > > Ok, but what's reasonable here? I think you're implying here that this small > change should go through a PEP process. > > So a question for the list, does this seemingly small change warrant a PEP? Note that the docs for callable() explicitly allow for false positives - callable(x) is true, but x() fails. See https://docs.python.org/3/library/functions.html#callable ("If this returns true, it is still possible that a call fails"). The "bug" you mentioned earlier in the thread is precisely that - a case where callable() returns True but calling the object fails: >It doesn't really work because of the incomplete checks done in `callable_builtin`. This is what I've tried: > >> >>> class DynamicCallable: >> ... is_callable = True >> ... >> ... def __init__(self, target): >> ... self.target = target >> ... >> ... @property >> ... def __call__(self): >> ... if self.is_callable: >> ... return self.target >> ... else: >> ... raise AttributeError("Not really ...") >> ... >> >>> dc = DynamicCallable(print) >> >>> dc(1, 2, 3) >> 1 2 3 >> >>> callable(dc) >> True >> >>> dc.is_callable = False >> >>> callable(dc) >> True ###### This should be False :( > > > If the "bug" is fixed, then the last thing in the above example would return False.. So I'm not clear what "small change" you're referring to here: * A change to CPython to reduce the number of false positives by making this case return False? If that, then no, I don't think a PEP is needed. But note that user code still can't assume that the above behaviour couldn't still happen in *other* cases, so the change would be of limited value (and whether it gets accepted depends on whether the complexity is justified by the benefit). * A change to the definition of callable() to remove the possibility of false positives at all? In that case, yes, a PEP probably *is* needed, as that's going to affect an awful lot of corner cases, and will impact all implementations. It's probably not correct to call this a "small change". * Something else? Paul From contact at ionelmc.ro Sun Apr 19 02:30:18 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sun, 19 Apr 2015 03:30:18 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sun, Apr 19, 2015 at 2:09 AM, Paul Moore wrote: > So I'm not clear what "small change" you're referring to here: > > * A change to CPython to reduce the number of false positives by > making this case return False? If that, then no, I don't think a PEP > is needed. But note that user code still can't assume that the above > behaviour couldn't still happen in *other* cases, so the change would > be of limited value (and whether it gets accepted depends on whether > the complexity is justified by the benefit). > * A change to the definition of callable() to remove the possibility > of false positives at all? In that case, yes, a PEP probably *is* > needed, as that's going to affect an awful lot of corner cases, and > will impact all implementations. It's probably not correct to call > this a "small change". > * Something else? > ?The discussion was about this small change: int PyCallable_Check(PyObject *x) { - if (x == NULL) + if (x == NULL) { return 0; - return x->ob_type->tp_call != NULL; + } + + return Py_TYPE(x)->tp_call && _PyObject_HasAttrId(x, &PyId___call__); } There are more explanations in the thread in case you want to know more. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Sun Apr 19 02:37:47 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sun, 19 Apr 2015 03:37:47 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: Sorry, I pasted a patch with changes unrelated to this discussion. This is the gist of the proposed change: int PyCallable_Check(PyObject *x) { if (x == NULL) return 0; - return x->ob_type->tp_call; + return x->ob_type->tp_call && _PyObject_HasAttrId(x, &PyId___call__); } Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Sun, Apr 19, 2015 at 3:30 AM, Ionel Cristian M?rie? wrote: > > On Sun, Apr 19, 2015 at 2:09 AM, Paul Moore wrote: > >> So I'm not clear what "small change" you're referring to here: >> >> * A change to CPython to reduce the number of false positives by >> making this case return False? If that, then no, I don't think a PEP >> is needed. But note that user code still can't assume that the above >> behaviour couldn't still happen in *other* cases, so the change would >> be of limited value (and whether it gets accepted depends on whether >> the complexity is justified by the benefit). >> * A change to the definition of callable() to remove the possibility >> of false positives at all? In that case, yes, a PEP probably *is* >> needed, as that's going to affect an awful lot of corner cases, and >> will impact all implementations. It's probably not correct to call >> this a "small change". >> * Something else? >> > > ?The discussion was about this small change: > > int > PyCallable_Check(PyObject *x) > { > - if (x == NULL) > + if (x == NULL) { > return 0; > - return x->ob_type->tp_call != NULL; > + } > + > + return Py_TYPE(x)->tp_call && _PyObject_HasAttrId(x, &PyId___call__); > } > > There are more explanations in the thread in case you want to know more. > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Apr 19 04:27:15 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 18 Apr 2015 19:27:15 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <553296C4.3060603@gmail.com> References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> Message-ID: On Apr 18, 2015, at 10:39, Yury Selivanov wrote: > > Hi Steven, > >> On 2015-04-18 1:11 PM, Steven D'Aprano wrote: >> Nicely written and very complete! I only have a couple of little >> niggles, which some might consider bike-shedding. See below. > > Thanks a lot! > >> >> >> On Fri, Apr 17, 2015 at 02:58:58PM -0400, Yury Selivanov wrote: >> >> [...] >>> From this point in this document we use the word *coroutine* to refer to >>> functions declared using the new syntax. *generator-based coroutine* is >>> used where necessary to refer to coroutines that are based on >>> generator syntax. >> It concerns me that we will have two kinds of coroutines. >> >> At the moment, we have two kinds of generator functions: those that are >> used as generators, and those that are used as coroutines, depending on >> whether they contain "yield expr" or "x = (yield expr)". That >> distinction won't go away, but now we will have even more ambiguity of >> language: there will be generators which are functions: >> >> def gen(): yield 1 >> >> generators which are not functions but the result of calling a generator >> function: >> >> g = gen() >> >> coroutines which are generators: >> >> def gen(): >> x = (yield 1) >> y = (yield x+1) >> >> cr = gen() >> >> and coroutines which are not generators: >> >> async def g(): >> await spam() >> >> Except that they will be implemented internally as generators. There >> will be generators that use yield and generators which give a syntax >> error with yield. My head hurts :-) >> >> I think it will be hard to unambiguously talk about these things without >> confusion, especially for people with limited asyncronous experience. >> >> I don't have an answer to this, but I know that even the generator >> versus generator-function ambiguity (both get called "generator") >> sometimes leads to confusion. (E.g. below you talk about passing a >> generator to types.async_def but it isn't clear to me whether you mean >> the generator or the generator function.) There's been at least one >> thread on python-list in the last few months about that. Is it possible >> to use another name to distinguish generator-based coroutines from >> async-based coroutines? >> >> It's a pity we can't steal the name "goroutine" from Go. I've always >> liked that name :-) but as I understand it, goroutines completely >> unrelated to coroutines. Hmmm... maybe we could use pyroutines? *wink* >> >> If the name must stay as-is, it would help to have an unambigous and >> clear summary of the various generator-or-coroutine uses and how they >> differ. > > The PEP kind of addresses this, but calling current approach > to do coroutines "generator-based coroutines", and the new > ones as just "coroutines". The only problem is that, when it's important to specify that you mean the new kind of coroutines explicitly, there's no obvious way to do so--if I say "async coroutines" someone may reasonably think I'm referring to "coroutines as user in asyncio". I could say "PEP XXX coroutines" or "awaitable coroutines" or "non-yield-from coroutines" or probably various other things, but none of them are the first thing that comes to mind. It definitely seems plausible that eventually the ambiguity will vanish, as nobody will talk about the other kind of coroutines except as an implementation detail for backporting code to 3.3/3.4 (which will itself probably not be common for that long) or porting the feature itself to other implementations. But it might be nice to have an official name that can be used when necessary until that happens. (I don't know what that name should be.) > The internal implementation based on generators is something > that was absolutely required to cover in the PEP in a great detail, > but almost everybody will forget about that fact pretty soon. > > The point of this PEP is to actually avoid any confusion, and > to just have two separate concepts: coroutines and generators > (the latter can still be used as a "coroutine-like" object, > but hopefully nobody will do that at some point). > >> >> >>> types.async_def() >>> ----------------- >>> >>> A new function ``async_def(gen)`` is added to the ``types`` module. It >>> applies ``CO_ASYNC`` flag to the passed generator's code object, so that it >>> returns a *coroutine object* when called. >>> >>> This feature enables an easy upgrade path for existing libraries. >> When you say "the passed generator", do you mean the generator function or the generator object? >> E.g. given: >> >> def gen(): >> yield 1 >> >> which do I use? async_def(gen()) or async_def(gen)? > > It's generator-function. It's a very good catch. > > I've updated the PEP. > > > Thanks! > Yury > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From piotr.jerzy.jurkiewicz at gmail.com Sun Apr 19 05:19:21 2015 From: piotr.jerzy.jurkiewicz at gmail.com (Piotr Jurkiewicz) Date: Sun, 19 Apr 2015 05:19:21 +0200 Subject: [Python-ideas] async/await in Python Message-ID: <55331EB9.8040007@gmail.com> 1. Overall I like the proposal very much. However, I have got one semantic remark. You propose `async for` as a syntax for asynchronous iterators: async for row in Cursor(): print(row) Wouldn't it be more semantically correct to use `await for` instead of `async for`? await for row in Cursor(): print(row) For me the word 'await' is an indicator that I am awaiting for some value being returned. For example, with simple `await` expression I am awaiting for a data being fetched from db: data = await db.fetch('SELECT ...') When I use asynchronous iterator I am awaiting for a value being returned as well. For example I am awaiting (in each iteration) for a row from a cursor. Therefore, it seems to me to be natural to use word 'await' instead of 'async'. Furthermore syntax 'await for row in cursor' reassembles natural English language. On the other hand, when I use context manager, I am not awaiting for any value, so syntax `async with` seems to be proper in that case: async with session.transaction(): ... await session.update(data) Dart, for example, goes that way. They use `await` expression for awaiting single Future and `await for` statement for asynchronous iterators: await for (variable declaration in expression) { // Executes each time the stream emits a value. } 2. I would like to go little beyond this proposal and think about composition of async coroutines (aka waiting for multiple coroutines). For example C# has helper functions WhenAll and WhenAny for that: await Task.WhenAll(tasks_list); await Task.WhenAny(tasks_list); In asyncio module there is a function asyncio.wait() which can be used to achieve similar result: asyncio.wait(fs, timeout=None, return_when=ALL_COMPLETED) asyncio.wait(fs, timeout=None, return_when=FIRST_COMPLETED) However, after introduction of `await` its name becomes problematic. First, it reassembles `await` too much and can cause a confusion. Second, its usage would result in an awkward 'await wait': done, pending = await asyncio.wait(coroutines_list) results = [] for task in done: results.append(task.result()) Another problem with asyncio.wait() is that it returns Tasks, not their results directly, so user has to unpack them. There is function asyncio.gather(*coros_or_futures) which return results list directly, however it can be only used for ALL_COMPLETED case. There is also a function asyncio.wait_for() which (unlike asyncio.wait()) unpacks the result, but can only be used for one coroutine (so what is the difference from `await` expression?). Finally, there is asyncio.as_completed() which returns iterator for iterating over coroutines results as they complete (but I don't know how exactly this iterator relates to async iterators proposed here). I can imagine the set of three functions being exposed to user to control waiting for multiple coroutines: asynctools.as_done() # returns asynchronous iterator for iterating over the results of coroutines as they complete asynctools.all_done() # returns a future aggregating results from the given coroutine objects, which awaited returns list of results (like asyncio.gather()) asynctools.any_done() # returns a future, which awaited returns result of first completed coroutine Example: from asynctools import as_done, all_done, any_done corobj0 = async_sql_query("SELECT...") corobj1 = async_memcached_get("someid") corobj2 = async_http_get("http://python.org") # ------------------------------------------------ # Iterate over results as coroutines complete # using async iterator await for result in as_done([corobj0, corobj1, corobj2]): print(result) # ------------------------------------------------ # Await for results of all coroutines # using async iterator results = [] await for result in as_done([corobj0, corobj1, corobj2]): results.append(result) # or using shorthand coroutine all_done() results = await all_done([corobj0, corobj1, corobj2]) # ------------------------------------------------ # Await for a result of first completed coroutine # using async iterator await for result in as_done([corobj0, corobj1, corobj2]): first_result = result break # or using shorthand coroutine any_done() first_result = await any_done([corobj0, corobj1, corobj2]) I deliberately placed these functions in a new asynctools module, not in the asyncio module. I find asyncio module being too much complicated to expose it to an ordinary user. There are four very similar concepts used in it: Coroutine (function), Coroutine (object), Future and Task. In addition many functions accept both coroutines and Futures in the same argument, Task is a subclass of Future -- it makes people very confused. It is difficult to grasp what are differences between them and how they relate to each other. For comparison in JavaScript that are only two concepts: async functions and Promises. (Furthermore, after this PEP being accepted there will be fifth concept: old-style coroutines. And there are also concurrent.futures.Futures...) Personally, I think that asyncio module should be refactored and broken into two separate modules, named for example: - asyncloop # consisting low-level loop-related things, mostly not intended to be used by the average user (apart from get_event_loop() and run_until_xxx()) - asynctools # consisting high-level helper functions, like described before As with this PEP async/await will become first class member of Python environment, all rest high-level functions should be in my opinion moved from asyncio to appropriate modules, like socket or subprocess. These are the places where users will be looking for them. For example: socket.socket.recv() socket.socket.recv_async() socket.socket.sendall() socket.socket.sendall_async() socket.getaddrinfo() socket.getaddrinfo_async() Finally, concurrent.futures should either be renamed to avoid the usage of word 'future', or be made compatible with async/await. I know that I went far beyond scope of this PEP, but I think that these are the issues which will pop up after acceptance of this PEP sooner or later. Finally, I remind about my proposal from the beginning of this email, to use `await for` instead of `async for` for asynchronous iterators. What's your opinion about that? Piotr From yselivanov.ml at gmail.com Sun Apr 19 06:08:09 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 00:08:09 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <55331EB9.8040007@gmail.com> References: <55331EB9.8040007@gmail.com> Message-ID: <55332A29.5090602@gmail.com> Hi Piotr, Thank you very much for your detailed feedback. Answers below: On 2015-04-18 11:19 PM, Piotr Jurkiewicz wrote: > 1. Overall I like the proposal very much. However, I have got one > semantic remark. You propose `async for` as a syntax for asynchronous > iterators: > > async for row in Cursor(): > print(row) > > Wouldn't it be more semantically correct to use `await for` instead of > `async for`? > > await for row in Cursor(): > print(row) I like that the current proposal is simple. You use 'await' keyword only to call a coroutine, and you use 'async' only as a modifier, i.e. 'async def' becomes a coroutine, 'async for' is an asynchronous iteration block, etc. The less confusion we have the better. > > For me the word 'await' is an indicator that I am awaiting for some > value being returned. For example, with simple `await` expression I am > awaiting for a data being fetched from db: > > data = await db.fetch('SELECT ...') > > When I use asynchronous iterator I am awaiting for a value being > returned as well. For example I am awaiting (in each iteration) for a > row from a cursor. Therefore, it seems to me to be natural to use word > 'await' instead of 'async'. Furthermore syntax 'await for row in > cursor' reassembles natural English language. To me it reads different. There is no value of 'for' statement in Python, it's a block of code. Hence we use 'async' to mark it as an asynchronous block of code. > > On the other hand, when I use context manager, I am not awaiting for > any value, so syntax `async with` seems to be proper in that case: > > async with session.transaction(): > ... > await session.update(data) > > Dart, for example, goes that way. They use `await` expression for > awaiting single Future and `await for` statement for asynchronous > iterators: > > await for (variable declaration in expression) { > // Executes each time the stream emits a value. > } > > 2. I would like to go little beyond this proposal and think about > composition of async coroutines (aka waiting for multiple coroutines). > For example C# has helper functions WhenAll and WhenAny for that: > > await Task.WhenAll(tasks_list); > await Task.WhenAny(tasks_list); > > In asyncio module there is a function asyncio.wait() which can be used > to achieve similar result: > > asyncio.wait(fs, timeout=None, return_when=ALL_COMPLETED) > asyncio.wait(fs, timeout=None, return_when=FIRST_COMPLETED) > > However, after introduction of `await` its name becomes problematic. > First, it reassembles `await` too much and can cause a confusion. > Second, its usage would result in an awkward 'await wait': > > done, pending = await asyncio.wait(coroutines_list) > results = [] > for task in done: > results.append(task.result()) > > Another problem with asyncio.wait() is that it returns Tasks, not > their results directly, so user has to unpack them. There is function > asyncio.gather(*coros_or_futures) which return results list directly, > however it can be only used for ALL_COMPLETED case. There is also a > function asyncio.wait_for() which (unlike asyncio.wait()) unpacks the > result, but can only be used for one coroutine (so what is the > difference from `await` expression?). Finally, there is > asyncio.as_completed() which returns iterator for iterating over > coroutines results as they complete (but I don't know how exactly this > iterator relates to async iterators proposed here). > > I can imagine the set of three functions being exposed to user to > control waiting for multiple coroutines: > > asynctools.as_done() # returns asynchronous iterator for iterating > over the results of coroutines as they complete > asynctools.all_done() # returns a future aggregating results from the > given coroutine objects, which awaited returns list of results (like > asyncio.gather()) > asynctools.any_done() # returns a future, which awaited returns result > of first completed coroutine > > Example: > > from asynctools import as_done, all_done, any_done > > corobj0 = async_sql_query("SELECT...") > corobj1 = async_memcached_get("someid") > corobj2 = async_http_get("http://python.org") > > # ------------------------------------------------ > > # Iterate over results as coroutines complete > # using async iterator > > await for result in as_done([corobj0, corobj1, corobj2]): > print(result) > > # ------------------------------------------------ > > # Await for results of all coroutines > # using async iterator > > results = [] > await for result in as_done([corobj0, corobj1, corobj2]): > results.append(result) > > # or using shorthand coroutine all_done() > > results = await all_done([corobj0, corobj1, corobj2]) > > # ------------------------------------------------ > > # Await for a result of first completed coroutine > # using async iterator > > await for result in as_done([corobj0, corobj1, corobj2]): > first_result = result > break > > # or using shorthand coroutine any_done() > > first_result = await any_done([corobj0, corobj1, corobj2]) > > I deliberately placed these functions in a new asynctools module, not > in the asyncio module. I find asyncio module being too much > complicated to expose it to an ordinary user. There are four very > similar concepts used in it: Coroutine (function), Coroutine (object), > Future and Task. In addition many functions accept both coroutines and > Futures in the same argument, Task is a subclass of Future -- it makes > people very confused. It is difficult to grasp what are differences > between them and how they relate to each other. For comparison in > JavaScript that are only two concepts: async functions and Promises. > > (Furthermore, after this PEP being accepted there will be fifth > concept: old-style coroutines. And there are also > concurrent.futures.Futures...) > > Personally, I think that asyncio module should be refactored and > broken into two separate modules, named for example: > > - asyncloop # consisting low-level loop-related things, mostly not > intended to be used by the average user (apart from get_event_loop() > and run_until_xxx()) > - asynctools # consisting high-level helper functions, like described > before > > As with this PEP async/await will become first class member of Python > environment, all rest high-level functions should be in my opinion > moved from asyncio to appropriate modules, like socket or subprocess. > These are the places where users will be looking for them. For example: > > socket.socket.recv() > socket.socket.recv_async() > socket.socket.sendall() > socket.socket.sendall_async() > socket.getaddrinfo() > socket.getaddrinfo_async() > > Finally, concurrent.futures should either be renamed to avoid the > usage of word 'future', or be made compatible with async/await. > > I know that I went far beyond scope of this PEP, but I think that > these are the issues which will pop up after acceptance of this PEP > sooner or later. You're exactly right--your ideas are very nice and sound,--but are outside of the scope of PEP 492. One step at a time. First, we do the syntax changes and integrate only the most important functions and builtins. Later, we get the feedback and integrate other important features to the standard library (for instance, I liked your idea about asyncloop module. I think it should be prototyped and put on PyPI if the PEP is accepted). Thanks a lot! Yury From yselivanov.ml at gmail.com Sun Apr 19 06:08:59 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 00:08:59 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <3455689F-F60F-48BA-ABBE-189CAE1F1E10@langa.pl> References: <553157F2.8060408@gmail.com> <3455689F-F60F-48BA-ABBE-189CAE1F1E10@langa.pl> Message-ID: <55332A5B.9000300@gmail.com> On 2015-04-17 10:09 PM, ?ukasz Langa wrote: > Obviously +1. Thank you, ?ukasz! Yury From yselivanov.ml at gmail.com Sun Apr 19 06:19:13 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 00:19:13 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> Message-ID: <55332CC1.6000501@gmail.com> Chris, On 2015-04-18 5:29 PM, Chris Angelico wrote: > On Sun, Apr 19, 2015 at 3:39 AM, Yury Selivanov wrote: >> The internal implementation based on generators is something >> that was absolutely required to cover in the PEP in a great detail, >> but almost everybody will forget about that fact pretty soon. >> >> The point of this PEP is to actually avoid any confusion, and >> to just have two separate concepts: coroutines and generators >> (the latter can still be used as a "coroutine-like" object, >> but hopefully nobody will do that at some point). > Which means that these are three very VERY different things, all of > which happen to be implemented (under the covers) using similar > mechanisms: > > async def corout(): > await spam() > > def gen(): > yield from [1,2,3] > > class iter: > def __init__(self): self.state = 0 > def __iter__(self): return self > def __next__(self): > if self.state == 3: raise StopIteration > self.state += 1 > return self.state > > Technically, a generator object is an iterator, and a coroutine is a > generator, but they're really implementation details. PEP 479 is all > about not expecting "raise StopIteration" to terminate gen(); in the > same way, programmers should not expect "for x in corout()" to do > anything meaningful, because they're completely different things. > > Have I understood this correctly? Yes, you're correct, 'for x in coroutine()' shouldn't do anything meaningful in the client code, but it's important to allow this in an event loop (or at least to allow .send() and .throw() methods of coroutine object). As of now, the PEP and reference implementation allow using iter() built-in and for..in statements on coroutine-objects. I was going to raise this question next week, when I gather the initial feedback (I have a branch where this restriction is implemented, so it's at least possible). Best, Yury From abarnert at yahoo.com Sun Apr 19 06:55:45 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 18 Apr 2015 21:55:45 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> Message-ID: On Apr 18, 2015, at 19:27, Andrew Barnert wrote: > >> On Apr 18, 2015, at 10:39, Yury Selivanov wrote: >> >> Hi Steven, >> >>> On 2015-04-18 1:11 PM, Steven D'Aprano wrote: >>> Nicely written and very complete! I only have a couple of little >>> niggles, which some might consider bike-shedding. See below. >> >> Thanks a lot! >> >>> >>> >>> On Fri, Apr 17, 2015 at 02:58:58PM -0400, Yury Selivanov wrote: >>> >>> [...] >>>> From this point in this document we use the word *coroutine* to refer to >>>> functions declared using the new syntax. *generator-based coroutine* is >>>> used where necessary to refer to coroutines that are based on >>>> generator syntax. >>> It concerns me that we will have two kinds of coroutines. >>> >>> At the moment, we have two kinds of generator functions: those that are >>> used as generators, and those that are used as coroutines, depending on >>> whether they contain "yield expr" or "x = (yield expr)". That >>> distinction won't go away, but now we will have even more ambiguity of >>> language: there will be generators which are functions: >>> >>> def gen(): yield 1 >>> >>> generators which are not functions but the result of calling a generator >>> function: >>> >>> g = gen() >>> >>> coroutines which are generators: >>> >>> def gen(): >>> x = (yield 1) >>> y = (yield x+1) >>> >>> cr = gen() >>> >>> and coroutines which are not generators: >>> >>> async def g(): >>> await spam() >>> >>> Except that they will be implemented internally as generators. There >>> will be generators that use yield and generators which give a syntax >>> error with yield. My head hurts :-) >>> >>> I think it will be hard to unambiguously talk about these things without >>> confusion, especially for people with limited asyncronous experience. >>> >>> I don't have an answer to this, but I know that even the generator >>> versus generator-function ambiguity (both get called "generator") >>> sometimes leads to confusion. (E.g. below you talk about passing a >>> generator to types.async_def but it isn't clear to me whether you mean >>> the generator or the generator function.) There's been at least one >>> thread on python-list in the last few months about that. Is it possible >>> to use another name to distinguish generator-based coroutines from >>> async-based coroutines? >>> >>> It's a pity we can't steal the name "goroutine" from Go. I've always >>> liked that name :-) but as I understand it, goroutines completely >>> unrelated to coroutines. Hmmm... maybe we could use pyroutines? *wink* >>> >>> If the name must stay as-is, it would help to have an unambigous and >>> clear summary of the various generator-or-coroutine uses and how they >>> differ. >> >> The PEP kind of addresses this, but calling current approach >> to do coroutines "generator-based coroutines", and the new >> ones as just "coroutines". > > The only problem is that, when it's important to specify that you mean the new kind of coroutines explicitly, there's no obvious way to do so--if I say "async coroutines" someone may reasonably think I'm referring to "coroutines as user in asyncio". Actually, to answer my question: I could just say "new-style coroutine". When I say "new-style class", novices aren't confused by that, and they get the idea that these are probably the kind of class they want; experts have already learned what it means; people in between can look it up easily enough if it's relevant. And of course it's obvious how to shorten "new-style class" when the difference isn't relevant to the discussion: you just say "class" (maybe with an asterisk saying "new-style, of course", but usually without even needing that). So, the same thing would probably work here. > I could say "PEP XXX coroutines" or "awaitable coroutines" or "non-yield-from coroutines" or probably various other things, but none of them are the first thing that comes to mind. > > It definitely seems plausible that eventually the ambiguity will vanish, as nobody will talk about the other kind of coroutines except as an implementation detail for backporting code to 3.3/3.4 (which will itself probably not be common for that long) or porting the feature itself to other implementations. > > But it might be nice to have an official name that can be used when necessary until that happens. (I don't know what that name should be.) > >> The internal implementation based on generators is something >> that was absolutely required to cover in the PEP in a great detail, >> but almost everybody will forget about that fact pretty soon. >> >> The point of this PEP is to actually avoid any confusion, and >> to just have two separate concepts: coroutines and generators >> (the latter can still be used as a "coroutine-like" object, >> but hopefully nobody will do that at some point). >> >>> >>> >>>> types.async_def() >>>> ----------------- >>>> >>>> A new function ``async_def(gen)`` is added to the ``types`` module. It >>>> applies ``CO_ASYNC`` flag to the passed generator's code object, so that it >>>> returns a *coroutine object* when called. >>>> >>>> This feature enables an easy upgrade path for existing libraries. >>> When you say "the passed generator", do you mean the generator function or the generator object? >>> E.g. given: >>> >>> def gen(): >>> yield 1 >>> >>> which do I use? async_def(gen()) or async_def(gen)? >> >> It's generator-function. It's a very good catch. >> >> I've updated the PEP. >> >> >> Thanks! >> Yury >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From greg.ewing at canterbury.ac.nz Sun Apr 19 08:21:59 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 19 Apr 2015 18:21:59 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> Message-ID: <55334987.3090108@canterbury.ac.nz> Andrew Barnert wrote: > The only problem is that, when it's important to specify that you mean the > new kind of coroutines explicitly, there's no obvious way to do so--if I say > "async coroutines" someone may reasonably think I'm referring to "coroutines > as user in asyncio". PEP 3152 solves this problem by calling them "cofunctions", which doesn't collide with any existing terminology that I'm aware of. -- Greg From abarnert at yahoo.com Sun Apr 19 09:23:29 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 19 Apr 2015 00:23:29 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <55334987.3090108@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> <55334987.3090108@canterbury.ac.nz> Message-ID: On Apr 18, 2015, at 23:21, Greg Ewing wrote: > > Andrew Barnert wrote: >> The only problem is that, when it's important to specify that you mean the >> new kind of coroutines explicitly, there's no obvious way to do so--if I say >> "async coroutines" someone may reasonably think I'm referring to "coroutines >> as user in asyncio". > > PEP 3152 solves this problem by calling them "cofunctions", > which doesn't collide with any existing terminology that > I'm aware of. Sure; if we go with the "cofunction" names, there's no conflict; if we go with the "coroutine" names, there is, although it may be easily avoidable. So that counts as a point in favor of the former. In the other direction, while don't think (although some people in this thread obviously disagree) there's anything inherently confusing about "cofunction", or anything inherently intuitive about "awaitable coroutine", the fact that F# used the latter terminology, and a number of well-known languages have copied it and heavily encouraged its use, definitely makes a difference. Anyway, since the difference is (as you pointed out) ultimately just about names (and maybe some minor details of which features go into the first version and which get added later), I think the opinions of to the people who will be teaching and/or evangelizing the feature are so much more useful than mine that I don't have any more to say. > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From p.f.moore at gmail.com Sun Apr 19 12:14:19 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 19 Apr 2015 11:14:19 +0100 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On 19 April 2015 at 01:37, Ionel Cristian M?rie? wrote: > Sorry, I pasted a patch with changes unrelated to this discussion. This is > the gist of the proposed change: > > int > PyCallable_Check(PyObject *x) > { > if (x == NULL) > return 0; > - return x->ob_type->tp_call; > + return x->ob_type->tp_call && _PyObject_HasAttrId(x, &PyId___call__); > } > OK, thanks. I'd say that doesn't need a PEP. But because it makes callable() a bit slower, and means that it can execute arbitrary Python code where it doesn't at the moment, for a pretty small gain (a few rare cases will no longer give a "false positive" result from callable(), but other false positives could well still exist), I can imagine the change being rejected based on the benefits not justifying the cost. Personally, I don't often use callable(). So I don't have a strong opinion. But when I *do* use it, it's because I want to check if something is callable *without* calling it - so the fact that callable() doesn't run arbitrary code is relevant to me. And the motivating use cases (__call__ being a property on the class object) seem pretty obscure - I've never encountered anything like that in real life. So overall, I'd be -0 on the change. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Sun Apr 19 13:28:32 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Sun, 19 Apr 2015 11:28:32 +0000 (UTC) Subject: [Python-ideas] with-except-finally blocks References: <552F5929.4030702@thekunderts.net> <20150416130913.GO5663@ando.pearwood.info> Message-ID: <646997798451135487.171203sturla.molden-gmail.com@news.gmane.org> Steven D'Aprano wrote: > This has two obvious, and slightly different, interpretations: > > with open('spam.txt') as file: > try: > print(file.read()) > except IOError: > print('No spam here...') > finally: > print('done.') > > > try: > with open('spam.txt') as file: > print(file.read()) > except IOError: > print('No spam here...') > finally: > print('done.') Sometimes I want the first, sometimes the second. And I expect a conscious mind-reading interpreter which gives me the one I want. If the interpreter in retrospect thinks it has given me the wrong one it should raise SorryError. Sturla From contact at ionelmc.ro Sun Apr 19 13:57:57 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sun, 19 Apr 2015 14:57:57 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Sun, Apr 19, 2015 at 1:14 PM, Paul Moore wrote: > so the fact that callable() doesn't run arbitrary code is relevant to me. ?_PyObject_HasAttrId(x, &PyId___call__) ??wouldn't actually do the call. What did you meant to say? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Apr 19 14:44:51 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 19 Apr 2015 13:44:51 +0100 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On 19 April 2015 at 12:57, Ionel Cristian M?rie? wrote: >> so the fact that callable() doesn't run arbitrary code is relevant to me. > > > _PyObject_HasAttrId(x, &PyId___call__) wouldn't actually do the call. What > did you meant to say? Wouldn't it run the get method of the descriptor? I thought that was the point of the change? That was the "arbitrary code" I was referring to (obviously it's well-known code for properties, but for a user-defined descriptor it could be anything). Paul From steve at pearwood.info Sun Apr 19 16:01:38 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 20 Apr 2015 00:01:38 +1000 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> Message-ID: <20150419140137.GC5663@ando.pearwood.info> On Sat, Apr 18, 2015 at 09:55:45PM -0700, Andrew Barnert wrote: > > The only problem is that, when it's important to specify that you > > mean the new kind of coroutines explicitly, there's no obvious way > > to do so--if I say "async coroutines" someone may reasonably think > > I'm referring to "coroutines as user in asyncio". > > Actually, to answer my question: I could just say "new-style coroutine". > > When I say "new-style class", novices aren't confused by that, and > they get the idea that these are probably the kind of class they want; > experts have already learned what it means; people in between can look > it up easily enough if it's relevant. -1 on "new-style". New-style classes have been around since Python 2.3 in 2003, which is twelve years ago. In what way are they still "new"? At least with new-style classes, the plan always was to deprecate classic classes, and in Python 3 the distinction is gone and "type" and "class" are, at least, synonyms. But here I understand that the generator-based coroutines and the async-def syntax coroutines will coexist forever. When Python is 40+ years old, will anyone care that generator-based coroutines were added a few releases before async-def coroutines? > And of course it's obvious how to shorten "new-style class" when the > difference isn't relevant to the discussion: you just say "class" > (maybe with an asterisk saying "new-style, of course", but usually > without even needing that). That doesn't apply here. New-style classes and classic classes are *almost* the same. The differences are subtle. But the differences between generator-based and async-def coroutines are quite considerable. Let's see if I have them all... Generator-based: - must use `yield` as an expression; - cannot use `await`; (?) - must be declared with a plain `def`; - must be primed before use; - use the send() method; Async-def: - must not use `yield` or `yield from`; - may use `await`; (?) - must be declared with `async def`; - don't need priming; - the send() method isn't used; (?) Apart from the underlying technology, they are nothing alike. Earlier, with my tongue planted firmly in my cheek, I suggested "pyroutines" as a pun on "goroutines" (which is of course a pun on coroutines). Silly as that was, I'd rather it than enshrine "new-style" and "old-style" again. -- Steve From yselivanov.ml at gmail.com Sun Apr 19 17:30:23 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 11:30:23 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419140137.GC5663@ando.pearwood.info> References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> <20150419140137.GC5663@ando.pearwood.info> Message-ID: <5533CA0F.4060403@gmail.com> Steven, On 2015-04-19 10:01 AM, Steven D'Aprano wrote: > Let's see if I have them all... > > Generator-based: > - must use `yield` as an expression; > - cannot use `await`; (?) > - must be declared with a plain `def`; > - must be primed before use; > - use the send() method; Correct, generator-based cannot use 'await'; > > Async-def: > - must not use `yield` or `yield from`; > - may use `await`; (?) > - must be declared with `async def`; > - don't need priming; > - the send() method isn't used; (?) And async-def can use await; and send()/throw() methods are used by asyncio and libraries. Yury From ethan at stoneleaf.us Sun Apr 19 17:51:51 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 19 Apr 2015 08:51:51 -0700 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: Message-ID: <20150419155151.GA2926@stoneleaf.us> On 04/19, Ionel Cristian M?rie? wrote: > On Sun, Apr 19, 2015 at 1:14 PM, Paul Moore wrote: >> so the fact that callable() doesn't run arbitrary code is relevant to me. > >_PyObject_HasAttrId(x, &PyId___call__) wouldn't actually do the call. > What did you meant to say? The __get__ method of the descriptor, whether of 'property' or something home-grown, is the "arbitrary code", and could conceivably be quite complex and/or slow -- especially if it has to hit the network to determine whether a proxy object is callable. -- ~Ethan~ From contact at ionelmc.ro Sun Apr 19 17:57:15 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Sun, 19 Apr 2015 18:57:15 +0300 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: <20150419155151.GA2926@stoneleaf.us> References: <20150419155151.GA2926@stoneleaf.us> Message-ID: On Sun, Apr 19, 2015 at 6:51 PM, Ethan Furman wrote: > The __get__ method of the descriptor, whether of 'property' or something > home-grown, is the "arbitrary code", and could conceivably be quite > complex and/or slow -- especially if it has to hit the network to determine > whether a proxy object is callable. > ?Well indeed. But any property could do evil stuff on attribute access (as opposed to a call), should the programmer choose so. I don't see how this is relevant.? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sun Apr 19 20:32:15 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 19 Apr 2015 20:32:15 +0200 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> Message-ID: On Apr 18, 2015 10:34 AM, "Ionel Cristian M?rie?" wrote: > ?It's hard to say what's intended or not. At best is speculation, as python-the-language does not have a specification. What we have now is made up from many contributions from many people. The language reference in the docs *is* the spec. Perhaps you mean some something else by "specification"? Are you looking for something from a technical commitee or similar? The spec for Python is defined by Guido and the Python committers. It is not exhaustive but it is authoritative. The python-dev list is where language discussions are driven, including clarifications where the language reference has insufficient information. The PEP process is used to make changes to the language. So keep all this in mind if you are coming from another language community. In the case of __call__, the language reference is sufficiently clear. I have other concerns and suggestions regarding changing callable. See them on the issue on the tracker. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Apr 19 20:36:47 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Apr 2015 20:36:47 +0200 Subject: [Python-ideas] async/await in Python References: <553157F2.8060408@gmail.com> Message-ID: <20150419203647.11c47b7e@fsol> On Fri, 17 Apr 2015 20:19:42 +0000 (UTC) Victor Stinner wrote: > They are almost > required to make asyncio usage easier. "async for" would help database ORMs > or the aiofiles project (run file I/O in threads). "async with" helps also ORMs. I don't understand how it would help ORMs in any way. ORMs run database requests transparently when the user requests an attribute, which is completely at odds with the approach of explicitly marking (either through "yield from" or "await") all potentially yielding operations. Regards Antoine. From yselivanov.ml at gmail.com Sun Apr 19 20:44:00 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 14:44:00 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419203647.11c47b7e@fsol> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> Message-ID: <5533F770.50409@gmail.com> Hi Antoine, On 2015-04-19 2:36 PM, Antoine Pitrou wrote: > On Fri, 17 Apr 2015 20:19:42 +0000 (UTC) > Victor Stinner > wrote: >> They are almost >> required to make asyncio usage easier. "async for" would help database ORMs >> or the aiofiles project (run file I/O in threads). "async with" helps also ORMs. > I don't understand how it would help ORMs in any way. ORMs run database > requests transparently when the user requests an attribute, which is > completely at odds with the approach of explicitly marking (either > through "yield from" or "await") all potentially yielding operations. It depends on the ORM. It's possible to completely describe the shape of the requested data; ORM then will query the data and lazily unpack it for the user with a nice object interface. Queries in __getattr__ are bad because of many reasons. What will really help database drivers, is the proposed 'async for' and 'async with'. 'async for' allows you to create a cursor that supports asynchronous iteration protocol, and can prefetch data to make iteration efficient. 'async with' enables to have sane transaction context managers, i.e. the ones that can actually commit changes in their '__aexit__' coroutine. Thanks, Yury From solipsis at pitrou.net Sun Apr 19 20:52:39 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Apr 2015 20:52:39 +0200 Subject: [Python-ideas] async/await in Python References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> Message-ID: <20150419205239.7f03909b@fsol> On Sun, 19 Apr 2015 14:44:00 -0400 Yury Selivanov wrote: > Queries in __getattr__ > are bad because of many reasons. If you think they are bad, it means you probably don't like ORMs, since that's a primary feature of theirs. Or perhaps you are confusing "ORM" with "database abstraction layer". They are not the same thing. Regards Antoine. From andrew.svetlov at gmail.com Sun Apr 19 20:57:59 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 19 Apr 2015 14:57:59 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419203647.11c47b7e@fsol> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> Message-ID: On Sun, Apr 19, 2015 at 2:36 PM, Antoine Pitrou wrote: > On Fri, 17 Apr 2015 20:19:42 +0000 (UTC) > Victor Stinner > wrote: >> They are almost >> required to make asyncio usage easier. "async for" would help database ORMs >> or the aiofiles project (run file I/O in threads). "async with" helps also ORMs. > > I don't understand how it would help ORMs in any way. ORMs run database > requests transparently when the user requests an attribute, which is > completely at odds with the approach of explicitly marking (either > through "yield from" or "await") all potentially yielding operations. > The PEP doesn't help ORMs from my perspective. But `async with` and `async for` are very useful for transactions (COMMIT or ROLLBACK on code block finish) and for iterations over cursor (server-side cursors are require for getting new data bulk on iteration). The same is for non-RDBMs also: Redis may produce data piece-by-piece on SCAN command for example. > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov From andrew.svetlov at gmail.com Sun Apr 19 21:00:43 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sun, 19 Apr 2015 15:00:43 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419205239.7f03909b@fsol> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> Message-ID: I think the PEP is not useful for ORMs but for low-level database drivers. They are may get benefits from `async with` and `async for` at least. On Sun, Apr 19, 2015 at 2:52 PM, Antoine Pitrou wrote: > On Sun, 19 Apr 2015 14:44:00 -0400 > Yury Selivanov > wrote: >> Queries in __getattr__ >> are bad because of many reasons. > > If you think they are bad, it means you probably don't like ORMs, since > that's a primary feature of theirs. > > Or perhaps you are confusing "ORM" with "database abstraction layer". > They are not the same thing. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov From yselivanov.ml at gmail.com Sun Apr 19 21:06:31 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 15:06:31 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419205239.7f03909b@fsol> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> Message-ID: <5533FCB7.9010401@gmail.com> Hi Antonie, On 2015-04-19 2:52 PM, Antoine Pitrou wrote: > On Sun, 19 Apr 2015 14:44:00 -0400 > Yury Selivanov > wrote: >> Queries in __getattr__ >> are bad because of many reasons. > If you think they are bad, it means you probably don't like ORMs, since > that's a primary feature of theirs. I don't like that particular "feature" of most popular ORMs available today. We have a fairly advanced ORM developed in-house in my firm that doesn't have this trait. (we are working hard to open source it soon) This is a slight off-topic, probably, but let me explain in a greater detail: In the bellow snippet of code we describe a shape of data that we want to fetch from the DB: BP = sprymix.content.blog.BlogPost posts = BP.filter(BP.is_visible == True).select([ BP.title, BP.label, BP.publication_date, BP.status, BP.is_public, (BP.header_image, [ BP.header_image.id ]), BP.body, BP.is_owned, (BP.owners, [ BP.owners.first_name, BP.owners.last_name ]) ]) Later, we can work with posts as with normal python object: iterate through it, access .title property and .owners collection etc. Everything is fetched in one shot. Now to modify something, you have to put your code in 'with transaction()' block, otherwise it's an error to write data. At the end of 'with transaction()' we commit the modified data. Now, implementing 'with' statement was possible for us only because we use greenlets, something that I don't want to use. Same goes for prefetching in 'for' loops -- only because we use greenlets we can do that. I'd really like to have some native syntax in python so that I can do posts = await BlogPost.select(...) And later work with asynchronous iteration/context managers. Thanks, Yury From solipsis at pitrou.net Sun Apr 19 21:23:11 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Apr 2015 21:23:11 +0200 Subject: [Python-ideas] async/await in Python References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> Message-ID: <20150419212311.31d7ab16@fsol> On Sun, 19 Apr 2015 15:06:31 -0400 Yury Selivanov wrote: > Hi Antonie, (OT: It's Antoine, not "Antonie") > In the bellow snippet of code we describe a shape of data > that we want to fetch from the DB: > > BP = sprymix.content.blog.BlogPost > posts = BP.filter(BP.is_visible == True).select([ > BP.title, > BP.label, > BP.publication_date, > BP.status, > BP.is_public, > (BP.header_image, [ > BP.header_image.id > ]), > BP.body, > BP.is_owned, > (BP.owners, [ > BP.owners.first_name, > BP.owners.last_name > ]) > ]) > > Later, we can work with posts as with normal python object: > iterate through it, access .title property and .owners > collection etc. Everything is fetched in one shot. Powerful ORMs such as SQLAlchemy can let you decide, at mapping definition time, which relations are fetched eagerly, which relations are fetched lazily, etc. The important point is that regular code then doesn't have to care: some attribute accesses run SQL queries, some don't, and it all happens behind the scenes. This means you can pass objects around without knowing up front which fields are going to be needed by consuming code. This "happens behind the scenes" aspect is what makes ORMs difficult to mix with explicit multi-threading strategies such as "await" and "yield from". Regards Antoine. From lukasz at langa.pl Sun Apr 19 21:28:07 2015 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 19 Apr 2015 12:28:07 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <5533FCB7.9010401@gmail.com> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> Message-ID: <53596A7F-A35D-4C14-B6B6-00E3356A3685@langa.pl> On Apr 19, 2015, at 12:06 PM, Yury Selivanov wrote: > On 2015-04-19 2:52 PM, Antoine Pitrou wrote: >> On Sun, 19 Apr 2015 14:44:00 -0400 >> Yury Selivanov >> wrote: >>> Queries in __getattr__ >>> are bad because of many reasons. >> If you think they are bad, it means you probably don't like ORMs, since >> that's a primary feature of theirs. > I don't like that particular "feature" of most popular ORMs > available today. Our personal likes aside, one of asyncio?s design goals was to disallow implicit yield points. This is a philosophical decision going back to PEP 20 and the attribute/method distinction in Python. As opposed to, say, Ruby and its Uniform Access Principle, in Python users expect obj.thing to be instant, as opposed to obj.get_thing(), which might take a non-trivial amount of time. In asyncio this is even more important since exclusively explicit yield points enable the user to reason about concurrency and thus the correctness of the implementation. As long as you?re not yielding, you are guaranteed* the global state of your thread is not going to mutate under your feet (signals and cancellations not-withstanding). Yes, it?s true that we can?t replicate the current SQLAlchemy API in a performant way in asyncio. As far as I understand, we don?t want to. That being said, you *can* implement ORMs with explicit DB communication. FWIW, it looks like it would be easier to modify the Django ORM?s API to work with asyncio. -- Lukasz Langa | Facebook Production Engineer | Global Consistency (+1) 650-681-7811 -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Sun Apr 19 21:29:53 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 15:29:53 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419212311.31d7ab16@fsol> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> <20150419212311.31d7ab16@fsol> Message-ID: <55340231.9010501@gmail.com> Hi Antoine, On 2015-04-19 3:23 PM, Antoine Pitrou wrote: > On Sun, 19 Apr 2015 15:06:31 -0400 > Yury Selivanov > wrote: >> Hi Antonie, > (OT: It's Antoine, not "Antonie") I apologize for misspelling your name. Sorry. Anyways, my only intent when I was replying to your last email was to show you, that it is possible to develop an ORM where queries do not occur at random places because of __getattr__/iteration. If you want to have ORM and asyncio and avoid using greenlets, then you have several problems to solve: 1. Queries in getattr -- that's possible it you always describe the shape of data you want. If you later access some attribute that wasn't fetched -- you get an error. 2. Sane syntax for transaction managers, session managers and connection pools. Of course you can use 'yield from', 'try' and 'finally' right now. 3. San syntax for iteration. You can use 'while True' but it's not elegant. Thank you, Yury From steve at pearwood.info Sun Apr 19 21:33:15 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 20 Apr 2015 05:33:15 +1000 Subject: [Python-ideas] Fix that broken callable builtin In-Reply-To: <20150418173857.GZ5663@ando.pearwood.info> References: <20150417211030.GE28035@stoneleaf.us> <20150418102633.GV5663@ando.pearwood.info> <20150418112332.GX5663@ando.pearwood.info> <20150418173857.GZ5663@ando.pearwood.info> Message-ID: <20150419193315.GD5663@ando.pearwood.info> On Sun, Apr 19, 2015 at 03:38:57AM +1000, Steven D'Aprano wrote: > However, I think *this* is a bug in callable: [snip examples] > I think callable() is badly broken. (Or at least *was* broken in 3.3 > -- I don't have 3.4 handy to try it.) Ah, the embarassment! It turned out that I had monkey-patched callable() and was calling the broken monkey-patched version, not the original. Sorry for the noise. -- Steve From nicholas.chammas at gmail.com Sun Apr 19 21:35:20 2015 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Sun, 19 Apr 2015 19:35:20 +0000 Subject: [Python-ideas] async/await in Python In-Reply-To: <55340231.9010501@gmail.com> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> <20150419212311.31d7ab16@fsol> <55340231.9010501@gmail.com> Message-ID: Is Mike Bayer on this list? Just curious as a n00b observer if he has thoughts on this new PEP, since we are talking about SQLAlchemy. Nick On Sun, Apr 19, 2015 at 3:30 PM Yury Selivanov wrote: > Hi Antoine, > > On 2015-04-19 3:23 PM, Antoine Pitrou wrote: > > On Sun, 19 Apr 2015 15:06:31 -0400 > > Yury Selivanov > > wrote: > >> Hi Antonie, > > (OT: It's Antoine, not "Antonie") > I apologize for misspelling your name. Sorry. > > > Anyways, my only intent when I was replying to your last > email was to show you, that it is possible to develop > an ORM where queries do not occur at random places > because of __getattr__/iteration. > > If you want to have ORM and asyncio and avoid using > greenlets, then you have several problems to solve: > > 1. Queries in getattr -- that's possible it you always > describe the shape of data you want. If you later > access some attribute that wasn't fetched -- you get > an error. > > 2. Sane syntax for transaction managers, session > managers and connection pools. Of course you can > use 'yield from', 'try' and 'finally' right now. > > 3. San syntax for iteration. You can use 'while True' > but it's not elegant. > > Thank you, > Yury > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Sun Apr 19 21:41:35 2015 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 19 Apr 2015 12:41:35 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> <20150419212311.31d7ab16@fsol> <55340231.9010501@gmail.com> Message-ID: <85F2B12D-538D-4978-A05E-2FF81FF2FB36@langa.pl> On Apr 19, 2015, at 12:35 PM, Nicholas Chammas wrote: > > Is Mike Bayer on this list? Just curious as a n00b observer if he has thoughts on this new PEP, since we are talking about SQLAlchemy. I don?t think he is but he did review the PEP. He pointed out the lack of an asynchronous __getattr__. -- Lukasz Langa | Facebook Production Engineer | Global Consistency (+1) 650-681-7811 -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Apr 19 21:42:17 2015 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Apr 2015 21:42:17 +0200 Subject: [Python-ideas] async/await in Python References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> <20150419212311.31d7ab16@fsol> <55340231.9010501@gmail.com> Message-ID: <20150419214217.4bb3d6da@fsol> On Sun, 19 Apr 2015 19:35:20 +0000 Nicholas Chammas wrote: > Is Mike Bayer > > on this list? Just curious as a n00b observer if he has thoughts on this > new PEP, since we are talking about SQLAlchemy. This new PEP doesn't bring anything new to the table on the topic. Mike Bayer's article is still perfectly applicable in a "await / async" world. Regards Antoine. From gmludo at gmail.com Mon Apr 20 00:03:08 2015 From: gmludo at gmail.com (Ludovic Gasc) Date: Mon, 20 Apr 2015 00:03:08 +0200 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419214217.4bb3d6da@fsol> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> <20150419212311.31d7ab16@fsol> <55340231.9010501@gmail.com> <20150419214217.4bb3d6da@fsol> Message-ID: Maybe we should forget ORM special use-case, and think more generally about the problem behind that: What's the problem ? It's impossible to use yield from / await on magic methods like __getattr__ or __init__ We could follow the same semantic as described in this PEP: __agetattr__ or __ainit__ But: 1. I've no idea if it's possible to implement that in CPython's internals and in others Python implementations. 2. At least to me, this feature should be another PEP, we have enough changes in this PEP. 3. Do we need to implement that ? Even if, personally, I don't use ORMs when I interact with databases, I've the feeling that this feature should be interesting, at least to show that async pattern is a first-class citizen in Python. -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ 2015-04-19 21:42 GMT+02:00 Antoine Pitrou : > On Sun, 19 Apr 2015 19:35:20 +0000 > Nicholas Chammas > wrote: > > Is Mike Bayer > > < > http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/> > > on this list? Just curious as a n00b observer if he has thoughts on this > > new PEP, since we are talking about SQLAlchemy. > > This new PEP doesn't bring anything new to the table on the topic. Mike > Bayer's article is still perfectly applicable in a "await / async" > world. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Apr 20 03:02:51 2015 From: greg.ewing at canterbury.ac.nz (Greg) Date: Mon, 20 Apr 2015 13:02:51 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419140137.GC5663@ando.pearwood.info> References: <553157F2.8060408@gmail.com> <20150418171140.GY5663@ando.pearwood.info> <553296C4.3060603@gmail.com> <20150419140137.GC5663@ando.pearwood.info> Message-ID: <5534503B.40101@canterbury.ac.nz> On 20/04/2015 2:01 a.m., Steven D'Aprano wrote: > Generator-based: > - must be primed before use; Not when you use it via yield-from. -- Greg From yselivanov.ml at gmail.com Mon Apr 20 05:10:13 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sun, 19 Apr 2015 23:10:13 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <5531A390.8060405@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> Message-ID: <55346E15.9040204@gmail.com> Hi Greg, I added a section about PEP 3152 to PEP 492: https://hg.python.org/peps/rev/428c7c753500 Please take a look, and feel free to correct me if I made any mistakes or forgot to specify something. Thank you, Yury On 2015-04-17 8:21 PM, Greg Ewing wrote: > Yury Selivanov wrote: >> >> Here's my proposal to add async/await in Python. > > You've essentially reinvented PEP 3152 - Cofunctions. > > https://www.python.org/dev/peps/pep-3152/ > > Here's a summary of the relationships between them: > > PEP 3152 Nearest equivalent in Yury's PEP > -------- -------------------------------- > > codef f(args): async def f(args): > > cocall f(args) await f(args) > > __cocall__ __await__ > > costart() async_def() > > There is currently no equivalent of "async for" and > "async with" in PEP 3152, but they could easily be added. > I would probably spell them "cofor" and "cowith". > > As the author of PEP 3152 I'm obviously biased, but I > think my spellings are more elegant and less disruptive > to reading of the code. > > PEP 3152 is currently marked as deferred. Maybe it's > time to revive it? If Yury's pep is to be considered, > we ought to discuss the relative merits of the two. > From abarnert at yahoo.com Mon Apr 20 05:22:38 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 20 Apr 2015 03:22:38 +0000 (UTC) Subject: [Python-ideas] async/await in Python In-Reply-To: <20150419140137.GC5663@ando.pearwood.info> References: <20150419140137.GC5663@ando.pearwood.info> Message-ID: <1322111806.23088.1429500158914.JavaMail.yahoo@mail.yahoo.com> On Sunday, April 19, 2015 7:02 AM, Steven D'Aprano wrote: > > On Sat, Apr 18, 2015 at 09:55:45PM -0700, Andrew Barnert wrote: > >> > The only problem is that, when it's important to specify that you >> > mean the new kind of coroutines explicitly, there's no obvious way > >> > to do so--if I say "async coroutines" someone may reasonably > think >> > I'm referring to "coroutines as user in asyncio". >> >> Actually, to answer my question: I could just say "new-style > coroutine". >> >> When I say "new-style class", novices aren't confused by > that, and >> they get the idea that these are probably the kind of class they want; >> experts have already learned what it means; people in between can look >> it up easily enough if it's relevant. > > -1 on "new-style". New-style classes have been around since Python 2.3 > > in 2003, which is twelve years ago. In what way are they still "new"? > > At least with new-style classes, the plan always was to deprecate > classic classes, and in Python 3 the distinction is gone and "type" > and > "class" are, at least, synonyms. But here I understand that the > generator-based coroutines and the async-def syntax coroutines will > coexist forever. Read the message I was replying to: > The point of this PEP is to actually avoid any confusion, and > to just have two separate concepts: coroutines and generators > (the latter can still be used as a "coroutine-like" object, > but hopefully nobody will do that at some point). So, while building coroutines out of generators may never be deprecated, they intention is that in practice they do _not_ coexist forever. Yuri did back off the "quite soon" wording he used earlier, acknowledging that code will probably need to maintain backward compat with 3.4 for a while--which is why the naming confusion is, at least in the short term, a problem (or so it seems to me). If you think that intention is either impossible or undesirable, that's a different story (and it might affect some of the other decisions that went into the PEP?). > When Python is 40+ years old, will anyone care that > generator-based coroutines were added a few releases before async-def > coroutines? If they're both being used side-by-side, sure. But again, that's not the intention by the time Python is 40 years old. If the only reason anyone has to learn about generator-based coroutines is to work on code from around 2015, at which point they have to learn that this is how we did things before we had syntactic support, then yes, the fact that they are "old-style" is clearly relevant. (Of course there will always be a couple of people who like playing with language features for its own sake who will either dig up Greg Ewing's posts that led to the asyncio design, or rediscover the idea that you could build coroutines out of generators for themselves. But those kind of people, we don't have to worry about confusing?) >> And of course it's obvious how to shorten "new-style class" > when the >> difference isn't relevant to the discussion: you just say > "class" >> (maybe with an asterisk saying "new-style, of course", but > usually >> without even needing that). > > That doesn't apply here. New-style classes and classic classes are > *almost* the same. The differences are subtle. But the differences > between generator-based and async-def coroutines are quite considerable. But none of those differences are relevant to someone who's never used the generator-based coroutines, never learned them, and will likely never need to. Also, notice that people using 2.7 can very easily accidentally create classic classes and then run into bugs; nobody using 4.6 will accidentally create a generator-based coroutine. People using 2.7 still have to use classic classes to work with major frameworks like the stdlib's Tkinter; that's unlikely to be true in 4.6. And, unlike classic classes, which were a fundamental part of Python for something like 6 years, generator-based coroutines will end up being a footnote if they were only used in one provisional framework for one major version. > Let's see if I have them all... > > Generator-based: > - must use `yield` as an expression; > - cannot use `await`; (?) > - must be declared with a plain `def`; > - must be primed before use; > - use the send() method; > > Async-def: > - must not use `yield` or `yield from`; > - may use `await`; (?) > - must be declared with `async def`; > - don't need priming; > - the send() method isn't used; (?) > > Apart from the underlying technology, they are nothing alike. Well, if they really are nothing alike, we shouldn't call them both coroutines. But I don't think that's true. What they're used for, and how they work, are fundamentally the same; just the details are different. They are both coroutines; there's the new, clean way to do coroutines, and the old way that still works but you don't want to use it, and if you do need to use it you'll have to learn all the details. > Earlier, with my tongue planted firmly in my cheek, I suggested > "pyroutines" as a pun on "goroutines" (which is of course a > pun on > coroutines). Greg Ewing's pointed out that the proposal is almost equivalent to PEP 3152 except for the naming; his PEP used "cofunction" and related terms instead of "coroutine", which definitely solves the problem. But I think it may be too drastic a solution. Especially since the same style of await-driven async programming is used in other major languages, and referred to in terms of coroutines. If we have to explain that "Yes, Python has coroutines just like C#, but we call them cofunctions, and we have these other things that are similar but different and not compatible that we call coroutines", I think the proposal loses a lot of its appeal. If it's necessary, it's necessary, but we shouldn't do it unless it really is. > Silly as that was, I'd rather it than enshrine > "new-style" > and "old-style" again. From greg.ewing at canterbury.ac.nz Mon Apr 20 06:48:03 2015 From: greg.ewing at canterbury.ac.nz (Greg) Date: Mon, 20 Apr 2015 16:48:03 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <1322111806.23088.1429500158914.JavaMail.yahoo@mail.yahoo.com> References: <20150419140137.GC5663@ando.pearwood.info> <1322111806.23088.1429500158914.JavaMail.yahoo@mail.yahoo.com> Message-ID: <55348503.1060705@canterbury.ac.nz> On 20/04/2015 3:22 p.m., Andrew Barnert wrote: > If we have to > explain that "Yes, Python has coroutines just like C#, but we call > them cofunctions, and we have these other things that are similar but > different and not compatible that we call coroutines" To my way of thinking, a cofunction is a particular way of implementing a coroutine, in the same way that a generator function is a particular way of implementing an iterator. In other words, it's not cofunction vs. coroutine; rather, cofunctions and generators are both kinds of coroutine. -- Greg From luciano at ramalho.org Mon Apr 20 10:17:09 2015 From: luciano at ramalho.org (Luciano Ramalho) Date: Mon, 20 Apr 2015 05:17:09 -0300 Subject: [Python-ideas] async/await in Python Message-ID: Yuri, thank you for writing PEP 492! Victor Stinner reviewed the asyncio chapter in my book, Fluent Python [1], and he saw how much I complained about the overloading of `def` to define objects as different as functions, generators and coroutines, and about `yield from` which doesn't say anything to me or to many others to whom I've talked. With `async def` and `await` you solve both these problems! Also, `async with` and `async for` are brilliant. For the first time ever coroutines will be real first class citizens in Python! [1] http://shop.oreilly.com/product/0636920032519.do Alas, the book [1] is entering final production so it won't cover Python 3.5, but I do hope to see what you propose in Python 3.5, and if there is a second edition, it will be a pleasure to explain `async` and `await`! I did have time to add a paragraph about PEP 492 in a Further reading section, citing your name as the author. Cheers, Luciano -- Luciano Ramalho | Author of Fluent Python (O'Reilly, 2015) | http://shop.oreilly.com/product/0636920032519.do | Professor em: http://python.pro.br | Twitter: @ramalhoorg From yselivanov.ml at gmail.com Mon Apr 20 15:13:29 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 20 Apr 2015 09:13:29 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: Message-ID: <5534FB79.3050104@gmail.com> Hi Luciano, Thank you for so positive feedback! Let's hope the PEP will be accepted in time! Yury On 2015-04-20 4:17 AM, Luciano Ramalho wrote: > Yuri, thank you for writing PEP 492! > > Victor Stinner reviewed the asyncio chapter in my book, Fluent Python > [1], and he saw how much I complained about the overloading of `def` > to define objects as different as functions, generators and > coroutines, and about `yield from` which doesn't say anything to me or > to many others to whom I've talked. > > With `async def` and `await` you solve both these problems! Also, > `async with` and `async for` are brilliant. For the first time ever > coroutines will be real first class citizens in Python! > > [1] http://shop.oreilly.com/product/0636920032519.do > > Alas, the book [1] is entering final production so it won't cover > Python 3.5, but I do hope to see what you propose in Python 3.5, and > if there is a second edition, it will be a pleasure to explain `async` > and `await`! > > I did have time to add a paragraph about PEP 492 in a Further reading > section, citing your name as the author. > > Cheers, > > Luciano > > From mal at egenix.com Mon Apr 20 15:18:28 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 20 Apr 2015 15:18:28 +0200 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <55346E15.9040204@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> Message-ID: <5534FCA4.2030909@egenix.com> Hi Yury, do you have any plans of also exposing an easy to use C API for async and await ? I think this would be useful to e.g. implement asynchronous database access. Most of those adapters are written in C and need so a C API would help a lot with this. A problem I ran to with e.g. gevent is that it's not easy to use the available async APIs from C, even less so for systems like the ODBC API which require the use of polling for async processing and which don't give you access to e.g. tap into long running parts of the communication (sockets, etc.) which gevent does at the Python level. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From andrew.svetlov at gmail.com Mon Apr 20 15:34:23 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Mon, 20 Apr 2015 16:34:23 +0300 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <5534FCA4.2030909@egenix.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> Message-ID: Marc-Andre Lemburg, do you have ideas for how C API should look like? Personally I have no idea for now (but I have never thought about too much). On Mon, Apr 20, 2015 at 4:18 PM, M.-A. Lemburg wrote: > Hi Yury, > > do you have any plans of also exposing an easy to use C API > for async and await ? > > I think this would be useful to e.g. implement asynchronous > database access. Most of those adapters are written in C and > need so a C API would help a lot with this. > > A problem I ran to with e.g. gevent is that it's not easy > to use the available async APIs from C, even less so for > systems like the ODBC API which require the use of polling > for async processing and which don't give you access to e.g. > tap into long running parts of the communication (sockets, etc.) > which gevent does at the Python level. > > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Apr 20 2015) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov From yselivanov.ml at gmail.com Mon Apr 20 15:40:00 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 20 Apr 2015 09:40:00 -0400 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <5534FCA4.2030909@egenix.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> Message-ID: <553501B0.10201@gmail.com> Hi Marc-Andre, On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: > Hi Yury, > > do you have any plans of also exposing an easy to use C API > for async and await ? > > I think this would be useful to e.g. implement asynchronous > database access. Most of those adapters are written in C and > need so a C API would help a lot with this. I think one way to have a convenient C API is to implement a Future-like object in C -- an object with __await__ method that should return an iterator, which should also be implemented in C and provide a way to attach callback or to *pipeline* C functions. This way C code should seamlessly integrate with asyncio. I doubt that we have time to include this kind of API in the PEP, but it can be developed as a separate library. Your thoughts? > > A problem I ran to with e.g. gevent is that it's not easy > to use the available async APIs from C, even less so for > systems like the ODBC API which require the use of polling > for async processing and which don't give you access to e.g. > tap into long running parts of the communication (sockets, etc.) > which gevent does at the Python level. > > Thanks, The other problem is that asyncio itself is written in Python, so it won't be easy to use it from C. I think it might be a good idea to speed up asyncio by implementing core event loop and selectors in C and to have an easy to use C API on top of that, but it's not in the scope of the PEP. Thanks, Yury From yselivanov.ml at gmail.com Mon Apr 20 16:02:29 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 20 Apr 2015 10:02:29 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <53596A7F-A35D-4C14-B6B6-00E3356A3685@langa.pl> References: <553157F2.8060408@gmail.com> <20150419203647.11c47b7e@fsol> <5533F770.50409@gmail.com> <20150419205239.7f03909b@fsol> <5533FCB7.9010401@gmail.com> <53596A7F-A35D-4C14-B6B6-00E3356A3685@langa.pl> Message-ID: <553506F5.6080502@gmail.com> Lukasz, On 2015-04-19 3:28 PM, ?ukasz Langa wrote: > On Apr 19, 2015, at 12:06 PM, Yury Selivanov wrote: > >> On 2015-04-19 2:52 PM, Antoine Pitrou wrote: >>> On Sun, 19 Apr 2015 14:44:00 -0400 >>> Yury Selivanov >>> wrote: >>>> Queries in __getattr__ >>>> are bad because of many reasons. >>> If you think they are bad, it means you probably don't like ORMs, since >>> that's a primary feature of theirs. >> I don't like that particular "feature" of most popular ORMs >> available today. > Our personal likes aside, one of asyncio?s design goals was to disallow implicit yield points. This is a philosophical decision going back to PEP 20 and the attribute/method distinction in Python. As opposed to, say, Ruby and its Uniform Access Principle, in Python users expect obj.thing to be instant, as opposed to obj.get_thing(), which might take a non-trivial amount of time. > > In asyncio this is even more important since exclusively explicit yield points enable the user to reason about concurrency and thus the correctness of the implementation. As long as you?re not yielding, you are guaranteed* the global state of your thread is not going to mutate under your feet (signals and cancellations not-withstanding). > > Yes, it?s true that we can?t replicate the current SQLAlchemy API in a performant way in asyncio. As far as I understand, we don?t want to. That being said, you *can* implement ORMs with explicit DB communication. FWIW, it looks like it would be easier to modify the Django ORM?s API to work with asyncio. Exactly. This really summarizes the discussion. Thank you, Yury From mal at egenix.com Mon Apr 20 16:49:24 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 20 Apr 2015 16:49:24 +0200 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <553501B0.10201@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> <553501B0.10201@gmail.com> Message-ID: <553511F4.3050204@egenix.com> On 20.04.2015 15:40, Yury Selivanov wrote: > Hi Marc-Andre, > > On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: >> Hi Yury, >> >> do you have any plans of also exposing an easy to use C API >> for async and await ? >> >> I think this would be useful to e.g. implement asynchronous >> database access. Most of those adapters are written in C and >> need so a C API would help a lot with this. > > I think one way to have a convenient C API is to implement > a Future-like object in C -- an object with __await__ method > that should return an iterator, which should also be > implemented in C and provide a way to attach callback > or to *pipeline* C functions. > > This way C code should seamlessly integrate with asyncio. > > I doubt that we have time to include this kind of API in > the PEP, but it can be developed as a separate library. > > Your thoughts? I'm just sketching here, since I don't have much experience with writing async code (I'm using threads and multiple processes instead) and it probably shows ;-) >From the C side of things, I think using callbacks and a way to register/unregister polling functions would be great to have. The C code could then run like this in polling mode: long_running_query(...) { /* Start long running operation... creating a handle for it */ PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval); PyAsync_ReturnAwait(); } mypolling_fct(int handle) { /* Check completion of operation */ if (completed) { /* Fetch results into value */ PyAsync_UnregisterPoll(mypolling_fct, handle); PyAsync_ReturnReady(value); } else PyAsync_ReturnContinue(); } and like this in callback mode: long_running_query(...) { /* Create a handle for a long running operation... */ PyAsync_RegisterCallback(myquery_done, handle, timeout); /* Start long running operation... */ PyAsync_ReturnAwait(); } myquery_done(int handle, PyObject *value) { PyAsync_UnregisterCallback(myquery_done, handle); PyAsync_ReturnReady(value); } Both mechanisms will likely have to move the long running operation into a separate (blocking) thread, unless there's also a way to register an event loop selector somewhere and the C lib your interfacing to provides this mechanism as well. >> A problem I ran to with e.g. gevent is that it's not easy >> to use the available async APIs from C, even less so for >> systems like the ODBC API which require the use of polling >> for async processing and which don't give you access to e.g. >> tap into long running parts of the communication (sockets, etc.) >> which gevent does at the Python level. >> >> Thanks, > > The other problem is that asyncio itself is written in > Python, so it won't be easy to use it from C. I think > it might be a good idea to speed up asyncio by implementing > core event loop and selectors in C and to have an easy > to use C API on top of that, but it's not in the scope > of the PEP. Agreed. Having the loop in C would make this easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Apr 20 16:55:49 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 20 Apr 2015 16:55:49 +0200 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> Message-ID: <55351375.4060305@egenix.com> On 20.04.2015 15:34, Andrew Svetlov wrote: > Marc-Andre Lemburg, do you have ideas for how C API should look like? > Personally I have no idea for now (but I have never thought about too > much). See my reply to Yury. I don't have any preference for a particular approach. As long as we can have an API that's easy to use in 80% of the use cases, I'd be happy :-) (and then hopefully cover the remaining 20% with something more complex but still possible). > On Mon, Apr 20, 2015 at 4:18 PM, M.-A. Lemburg wrote: >> Hi Yury, >> >> do you have any plans of also exposing an easy to use C API >> for async and await ? >> >> I think this would be useful to e.g. implement asynchronous >> database access. Most of those adapters are written in C and >> need so a C API would help a lot with this. >> >> A problem I ran to with e.g. gevent is that it's not easy >> to use the available async APIs from C, even less so for >> systems like the ODBC API which require the use of polling >> for async processing and which don't give you access to e.g. >> tap into long running parts of the communication (sockets, etc.) >> which gevent does at the Python level. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From yselivanov.ml at gmail.com Mon Apr 20 17:17:05 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 20 Apr 2015 11:17:05 -0400 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <553511F4.3050204@egenix.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> <553501B0.10201@gmail.com> <553511F4.3050204@egenix.com> Message-ID: <55351871.3040902@gmail.com> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote: > On 20.04.2015 15:40, Yury Selivanov wrote: >> Hi Marc-Andre, >> >> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: >>> Hi Yury, >>> >>> do you have any plans of also exposing an easy to use C API >>> for async and await ? >>> >>> I think this would be useful to e.g. implement asynchronous >>> database access. Most of those adapters are written in C and >>> need so a C API would help a lot with this. >> I think one way to have a convenient C API is to implement >> a Future-like object in C -- an object with __await__ method >> that should return an iterator, which should also be >> implemented in C and provide a way to attach callback >> or to *pipeline* C functions. >> >> This way C code should seamlessly integrate with asyncio. >> >> I doubt that we have time to include this kind of API in >> the PEP, but it can be developed as a separate library. >> >> Your thoughts? > I'm just sketching here, since I don't have much experience > with writing async code (I'm using threads and multiple > processes instead) and it probably shows ;-) > > From the C side of things, I think using callbacks > and a way to register/unregister polling functions > would be great to have. > > The C code could then run like this in polling mode: > > long_running_query(...) { > /* Start long running operation... creating a handle for it */ > PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval); > PyAsync_ReturnAwait(); > } > > mypolling_fct(int handle) { > /* Check completion of operation */ > if (completed) { > /* Fetch results into value */ > PyAsync_UnregisterPoll(mypolling_fct, handle); > PyAsync_ReturnReady(value); > } > else > PyAsync_ReturnContinue(); > } > > and like this in callback mode: > > long_running_query(...) { > /* Create a handle for a long running operation... */ > PyAsync_RegisterCallback(myquery_done, handle, timeout); > /* Start long running operation... */ > PyAsync_ReturnAwait(); > } > > myquery_done(int handle, PyObject *value) { > PyAsync_UnregisterCallback(myquery_done, handle); > PyAsync_ReturnReady(value); > } > > Both mechanisms will likely have to move the long running > operation into a separate (blocking) thread, unless there's also > a way to register an event loop selector somewhere and the C lib > your interfacing to provides this mechanism as well. > I see what you want to do and why now. However, I can't imagine how this could be implemented without asyncio event loop being implemented in C. Coroutines from PEP 492 aren't an indepenedant concept, they require a library/framework with an event loop to make them work. The only way to go right now is to get 'get_event_loop()' and 'loop.call_soon()' working in C using Python C API. From that point you can implement Future object in C, see 'asyncio/futures.py'; Future._schedule_callbacks, Future.set_result and Future.__iter__ methods in particular. With all that, you can easily return your future object from C methods and everything should work just fine. I know this isn't ideal, and we actually should start a separate discussion on how we can improve things on our asyncio C side. Thanks, Yury From guido at python.org Mon Apr 20 17:42:59 2015 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Apr 2015 08:42:59 -0700 Subject: [Python-ideas] async/await in Python In-Reply-To: <5531A390.8060405@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> Message-ID: On Fri, Apr 17, 2015 at 5:21 PM, Greg Ewing wrote: > Yury Selivanov wrote: > >> >> Here's my proposal to add async/await in Python. >> > > You've essentially reinvented PEP 3152 - Cofunctions. > > https://www.python.org/dev/peps/pep-3152/ > I *thought* Yury's proposal sounded familiar! :-) > Here's a summary of the relationships between them: > > PEP 3152 Nearest equivalent in Yury's PEP > -------- -------------------------------- > > codef f(args): async def f(args): > > cocall f(args) await f(args) > > __cocall__ __await__ > > costart() async_def() > > There is currently no equivalent of "async for" and > "async with" in PEP 3152, but they could easily be added. > I would probably spell them "cofor" and "cowith". > Unfortunately that's a lot of new reserved words, and they aren't even words. :-) I like the elegance of Yury's idea of prefixing various statements with "async". It also happens to easy introduction, since the lexer can be adapted (for a few releases at least) not to treat "async" as a reserved word unless followed by "def" etc. > As the author of PEP 3152 I'm obviously biased, but I > think my spellings are more elegant and less disruptive > to reading of the code. > As the BDFL I'm perhaps also biased (Yury went over early stages of his draft with me at PyCon) but I prefer his spellings. > PEP 3152 is currently marked as deferred. Maybe it's > time to revive it? If Yury's pep is to be considered, > we ought to discuss the relative merits of the two. I wonder if, instead of putting forward a competing proposal that is so similar, you could help Yury with PEP 492? I'm sure he'd happily take you as a co-author. I have one question about PEP 3152. IIUC, it states that cofunctions cannot be called except using cocall, and there seems to be syntax that enforces that the "argument" to cocall looks like a function call. This would rule out things that are normally always allowed in Python, if you have ``` return foo(args) ``` you can replace that with ``` r = foo(args) return r ``` But the way I read PEP 3152 the call has to occur syntactically in the cocall form, so ``` cocall foo(args) ``` cannot be replaced with ``` x = foo(args) cocall x ``` This ability is occasionally useful in asyncio, for example when creating a task, we can write ``` x = Task(foo(args)) ``` and then someone else can write ``` yield from x ``` Your cocall syntax is lacking such ability, since cocall insist on call syntax ("cocall x" is a syntax error). It would seem that in general your cocall is not a substitution for yield from when used with a future or task, and that looks like a real deficiency to me. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Apr 20 18:15:12 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 20 Apr 2015 18:15:12 +0200 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <55351871.3040902@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> <553501B0.10201@gmail.com> <553511F4.3050204@egenix.com> <55351871.3040902@gmail.com> Message-ID: <55352610.1070508@egenix.com> On 20.04.2015 17:17, Yury Selivanov wrote: > > On 2015-04-20 10:49 AM, M.-A. Lemburg wrote: >> On 20.04.2015 15:40, Yury Selivanov wrote: >>> Hi Marc-Andre, >>> >>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: >>>> Hi Yury, >>>> >>>> do you have any plans of also exposing an easy to use C API >>>> for async and await ? >>>> >>>> I think this would be useful to e.g. implement asynchronous >>>> database access. Most of those adapters are written in C and >>>> need so a C API would help a lot with this. >>> I think one way to have a convenient C API is to implement >>> a Future-like object in C -- an object with __await__ method >>> that should return an iterator, which should also be >>> implemented in C and provide a way to attach callback >>> or to *pipeline* C functions. >>> >>> This way C code should seamlessly integrate with asyncio. >>> >>> I doubt that we have time to include this kind of API in >>> the PEP, but it can be developed as a separate library. >>> >>> Your thoughts? >> I'm just sketching here, since I don't have much experience >> with writing async code (I'm using threads and multiple >> processes instead) and it probably shows ;-) >> >> From the C side of things, I think using callbacks >> and a way to register/unregister polling functions >> would be great to have. >> >> The C code could then run like this in polling mode: >> >> long_running_query(...) { >> /* Start long running operation... creating a handle for it */ >> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval); >> PyAsync_ReturnAwait(); >> } >> >> mypolling_fct(int handle) { >> /* Check completion of operation */ >> if (completed) { >> /* Fetch results into value */ >> PyAsync_UnregisterPoll(mypolling_fct, handle); >> PyAsync_ReturnReady(value); >> } >> else >> PyAsync_ReturnContinue(); >> } >> >> and like this in callback mode: >> >> long_running_query(...) { >> /* Create a handle for a long running operation... */ >> PyAsync_RegisterCallback(myquery_done, handle, timeout); >> /* Start long running operation... */ >> PyAsync_ReturnAwait(); >> } >> >> myquery_done(int handle, PyObject *value) { >> PyAsync_UnregisterCallback(myquery_done, handle); >> PyAsync_ReturnReady(value); >> } >> >> Both mechanisms will likely have to move the long running >> operation into a separate (blocking) thread, unless there's also >> a way to register an event loop selector somewhere and the C lib >> your interfacing to provides this mechanism as well. >> > > I see what you want to do and why now. > > However, I can't imagine how this could be implemented > without asyncio event loop being implemented in C. > Coroutines from PEP 492 aren't an indepenedant concept, > they require a library/framework with an event loop > to make them work. > > The only way to go right now is to get 'get_event_loop()' > and 'loop.call_soon()' working in C using Python C API. > > From that point you can implement Future object in C, > see 'asyncio/futures.py'; Future._schedule_callbacks, > Future.set_result and Future.__iter__ methods in > particular. > > With all that, you can easily return your future object > from C methods and everything should work just fine. > > I know this isn't ideal, and we actually should start a > separate discussion on how we can improve things on > our asyncio C side. I'm not expecting this to be ready anytime soon, just wanted to bring this up, since it may be something that has to be considered in the async/await API design early rather than later to not make it too complex to add later on. I'm not sure, but going through the Python API from C can potentially result in the code to reenter which sounds like it could cause some unwanted race conditions. A C API would avoid this. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From andrew.svetlov at gmail.com Mon Apr 20 20:16:29 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Mon, 20 Apr 2015 21:16:29 +0300 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <55352610.1070508@egenix.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> <553501B0.10201@gmail.com> <553511F4.3050204@egenix.com> <55351871.3040902@gmail.com> <55352610.1070508@egenix.com> Message-ID: Marc-Andre, I see your need but we should write C Accelerators for asyncio first. The task is not related to the PEP directly, while I like to working on it soon. The problem is: accelerator should be backward compatibility and optional (no problem with it). But making C API requires much more: we need to implement the most critical parts of asyncio in C and then invent C API for that. Anyway, thank you for inspiration for desired C Async API. On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg wrote: > On 20.04.2015 17:17, Yury Selivanov wrote: >> >> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote: >>> On 20.04.2015 15:40, Yury Selivanov wrote: >>>> Hi Marc-Andre, >>>> >>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: >>>>> Hi Yury, >>>>> >>>>> do you have any plans of also exposing an easy to use C API >>>>> for async and await ? >>>>> >>>>> I think this would be useful to e.g. implement asynchronous >>>>> database access. Most of those adapters are written in C and >>>>> need so a C API would help a lot with this. >>>> I think one way to have a convenient C API is to implement >>>> a Future-like object in C -- an object with __await__ method >>>> that should return an iterator, which should also be >>>> implemented in C and provide a way to attach callback >>>> or to *pipeline* C functions. >>>> >>>> This way C code should seamlessly integrate with asyncio. >>>> >>>> I doubt that we have time to include this kind of API in >>>> the PEP, but it can be developed as a separate library. >>>> >>>> Your thoughts? >>> I'm just sketching here, since I don't have much experience >>> with writing async code (I'm using threads and multiple >>> processes instead) and it probably shows ;-) >>> >>> From the C side of things, I think using callbacks >>> and a way to register/unregister polling functions >>> would be great to have. >>> >>> The C code could then run like this in polling mode: >>> >>> long_running_query(...) { >>> /* Start long running operation... creating a handle for it */ >>> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval); >>> PyAsync_ReturnAwait(); >>> } >>> >>> mypolling_fct(int handle) { >>> /* Check completion of operation */ >>> if (completed) { >>> /* Fetch results into value */ >>> PyAsync_UnregisterPoll(mypolling_fct, handle); >>> PyAsync_ReturnReady(value); >>> } >>> else >>> PyAsync_ReturnContinue(); >>> } >>> >>> and like this in callback mode: >>> >>> long_running_query(...) { >>> /* Create a handle for a long running operation... */ >>> PyAsync_RegisterCallback(myquery_done, handle, timeout); >>> /* Start long running operation... */ >>> PyAsync_ReturnAwait(); >>> } >>> >>> myquery_done(int handle, PyObject *value) { >>> PyAsync_UnregisterCallback(myquery_done, handle); >>> PyAsync_ReturnReady(value); >>> } >>> >>> Both mechanisms will likely have to move the long running >>> operation into a separate (blocking) thread, unless there's also >>> a way to register an event loop selector somewhere and the C lib >>> your interfacing to provides this mechanism as well. >>> >> >> I see what you want to do and why now. >> >> However, I can't imagine how this could be implemented >> without asyncio event loop being implemented in C. >> Coroutines from PEP 492 aren't an indepenedant concept, >> they require a library/framework with an event loop >> to make them work. >> >> The only way to go right now is to get 'get_event_loop()' >> and 'loop.call_soon()' working in C using Python C API. >> >> From that point you can implement Future object in C, >> see 'asyncio/futures.py'; Future._schedule_callbacks, >> Future.set_result and Future.__iter__ methods in >> particular. >> >> With all that, you can easily return your future object >> from C methods and everything should work just fine. >> >> I know this isn't ideal, and we actually should start a >> separate discussion on how we can improve things on >> our asyncio C side. > > I'm not expecting this to be ready anytime soon, > just wanted to bring this up, since it may be something > that has to be considered in the async/await API design > early rather than later to not make it too complex to > add later on. > > I'm not sure, but going through the Python API from C > can potentially result in the code to reenter which sounds > like it could cause some unwanted race conditions. A C API > would avoid this. > > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Apr 20 2015) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov From mal at egenix.com Mon Apr 20 20:46:28 2015 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 20 Apr 2015 20:46:28 +0200 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> <553501B0.10201@gmail.com> <553511F4.3050204@egenix.com> <55351871.3040902@gmail.com> <55352610.1070508@egenix.com> Message-ID: <55354984.8030503@egenix.com> On 20.04.2015 20:16, Andrew Svetlov wrote: > Marc-Andre, I see your need but we should write C Accelerators for > asyncio first. > The task is not related to the PEP directly, while I like to working on it soon. > The problem is: accelerator should be backward compatibility and > optional (no problem with it). > But making C API requires much more: we need to implement the most > critical parts of asyncio in C and then invent C API for that. > Anyway, thank you for inspiration for desired C Async API. Understood, and thanks for pushing all this. Looking forward to some future PEP which outlines the C API for async/await :-) BTW: If you need some financial backup for these things, please consider writing a grant application to the PSF: https://www.python.org/psf/grants/ Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 20 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ > On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg wrote: >> On 20.04.2015 17:17, Yury Selivanov wrote: >>> >>> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote: >>>> On 20.04.2015 15:40, Yury Selivanov wrote: >>>>> Hi Marc-Andre, >>>>> >>>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: >>>>>> Hi Yury, >>>>>> >>>>>> do you have any plans of also exposing an easy to use C API >>>>>> for async and await ? >>>>>> >>>>>> I think this would be useful to e.g. implement asynchronous >>>>>> database access. Most of those adapters are written in C and >>>>>> need so a C API would help a lot with this. >>>>> I think one way to have a convenient C API is to implement >>>>> a Future-like object in C -- an object with __await__ method >>>>> that should return an iterator, which should also be >>>>> implemented in C and provide a way to attach callback >>>>> or to *pipeline* C functions. >>>>> >>>>> This way C code should seamlessly integrate with asyncio. >>>>> >>>>> I doubt that we have time to include this kind of API in >>>>> the PEP, but it can be developed as a separate library. >>>>> >>>>> Your thoughts? >>>> I'm just sketching here, since I don't have much experience >>>> with writing async code (I'm using threads and multiple >>>> processes instead) and it probably shows ;-) >>>> >>>> From the C side of things, I think using callbacks >>>> and a way to register/unregister polling functions >>>> would be great to have. >>>> >>>> The C code could then run like this in polling mode: >>>> >>>> long_running_query(...) { >>>> /* Start long running operation... creating a handle for it */ >>>> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval); >>>> PyAsync_ReturnAwait(); >>>> } >>>> >>>> mypolling_fct(int handle) { >>>> /* Check completion of operation */ >>>> if (completed) { >>>> /* Fetch results into value */ >>>> PyAsync_UnregisterPoll(mypolling_fct, handle); >>>> PyAsync_ReturnReady(value); >>>> } >>>> else >>>> PyAsync_ReturnContinue(); >>>> } >>>> >>>> and like this in callback mode: >>>> >>>> long_running_query(...) { >>>> /* Create a handle for a long running operation... */ >>>> PyAsync_RegisterCallback(myquery_done, handle, timeout); >>>> /* Start long running operation... */ >>>> PyAsync_ReturnAwait(); >>>> } >>>> >>>> myquery_done(int handle, PyObject *value) { >>>> PyAsync_UnregisterCallback(myquery_done, handle); >>>> PyAsync_ReturnReady(value); >>>> } >>>> >>>> Both mechanisms will likely have to move the long running >>>> operation into a separate (blocking) thread, unless there's also >>>> a way to register an event loop selector somewhere and the C lib >>>> your interfacing to provides this mechanism as well. >>>> >>> >>> I see what you want to do and why now. >>> >>> However, I can't imagine how this could be implemented >>> without asyncio event loop being implemented in C. >>> Coroutines from PEP 492 aren't an indepenedant concept, >>> they require a library/framework with an event loop >>> to make them work. >>> >>> The only way to go right now is to get 'get_event_loop()' >>> and 'loop.call_soon()' working in C using Python C API. >>> >>> From that point you can implement Future object in C, >>> see 'asyncio/futures.py'; Future._schedule_callbacks, >>> Future.set_result and Future.__iter__ methods in >>> particular. >>> >>> With all that, you can easily return your future object >>> from C methods and everything should work just fine. >>> >>> I know this isn't ideal, and we actually should start a >>> separate discussion on how we can improve things on >>> our asyncio C side. >> >> I'm not expecting this to be ready anytime soon, >> just wanted to bring this up, since it may be something >> that has to be considered in the async/await API design >> early rather than later to not make it too complex to >> add later on. >> >> I'm not sure, but going through the Python API from C >> can potentially result in the code to reenter which sounds >> like it could cause some unwanted race conditions. A C API >> would avoid this. >> >> Thanks, >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Apr 20 2015) >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > From gmludo at gmail.com Mon Apr 20 21:39:06 2015 From: gmludo at gmail.com (Ludovic Gasc) Date: Mon, 20 Apr 2015 21:39:06 +0200 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: <55354984.8030503@egenix.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> <553501B0.10201@gmail.com> <553511F4.3050204@egenix.com> <55351871.3040902@gmail.com> <55352610.1070508@egenix.com> <55354984.8030503@egenix.com> Message-ID: +1 for C Accelerators for AsyncIO. I don't have the level to implement that, nevertheless I can contribute freely with two other manners: benchmarks and launch that on several daemons to track bugs. -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ 2015-04-20 20:46 GMT+02:00 M.-A. Lemburg : > On 20.04.2015 20:16, Andrew Svetlov wrote: > > Marc-Andre, I see your need but we should write C Accelerators for > > asyncio first. > > The task is not related to the PEP directly, while I like to working on > it soon. > > The problem is: accelerator should be backward compatibility and > > optional (no problem with it). > > But making C API requires much more: we need to implement the most > > critical parts of asyncio in C and then invent C API for that. > > Anyway, thank you for inspiration for desired C Async API. > > Understood, and thanks for pushing all this. Looking forward to > some future PEP which outlines the C API for async/await :-) > > BTW: If you need some financial backup for these things, please > consider writing a grant application to the PSF: > > https://www.python.org/psf/grants/ > > Cheers, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Apr 20 2015) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > > > On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg wrote: > >> On 20.04.2015 17:17, Yury Selivanov wrote: > >>> > >>> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote: > >>>> On 20.04.2015 15:40, Yury Selivanov wrote: > >>>>> Hi Marc-Andre, > >>>>> > >>>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: > >>>>>> Hi Yury, > >>>>>> > >>>>>> do you have any plans of also exposing an easy to use C API > >>>>>> for async and await ? > >>>>>> > >>>>>> I think this would be useful to e.g. implement asynchronous > >>>>>> database access. Most of those adapters are written in C and > >>>>>> need so a C API would help a lot with this. > >>>>> I think one way to have a convenient C API is to implement > >>>>> a Future-like object in C -- an object with __await__ method > >>>>> that should return an iterator, which should also be > >>>>> implemented in C and provide a way to attach callback > >>>>> or to *pipeline* C functions. > >>>>> > >>>>> This way C code should seamlessly integrate with asyncio. > >>>>> > >>>>> I doubt that we have time to include this kind of API in > >>>>> the PEP, but it can be developed as a separate library. > >>>>> > >>>>> Your thoughts? > >>>> I'm just sketching here, since I don't have much experience > >>>> with writing async code (I'm using threads and multiple > >>>> processes instead) and it probably shows ;-) > >>>> > >>>> From the C side of things, I think using callbacks > >>>> and a way to register/unregister polling functions > >>>> would be great to have. > >>>> > >>>> The C code could then run like this in polling mode: > >>>> > >>>> long_running_query(...) { > >>>> /* Start long running operation... creating a handle for it */ > >>>> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval); > >>>> PyAsync_ReturnAwait(); > >>>> } > >>>> > >>>> mypolling_fct(int handle) { > >>>> /* Check completion of operation */ > >>>> if (completed) { > >>>> /* Fetch results into value */ > >>>> PyAsync_UnregisterPoll(mypolling_fct, handle); > >>>> PyAsync_ReturnReady(value); > >>>> } > >>>> else > >>>> PyAsync_ReturnContinue(); > >>>> } > >>>> > >>>> and like this in callback mode: > >>>> > >>>> long_running_query(...) { > >>>> /* Create a handle for a long running operation... */ > >>>> PyAsync_RegisterCallback(myquery_done, handle, timeout); > >>>> /* Start long running operation... */ > >>>> PyAsync_ReturnAwait(); > >>>> } > >>>> > >>>> myquery_done(int handle, PyObject *value) { > >>>> PyAsync_UnregisterCallback(myquery_done, handle); > >>>> PyAsync_ReturnReady(value); > >>>> } > >>>> > >>>> Both mechanisms will likely have to move the long running > >>>> operation into a separate (blocking) thread, unless there's also > >>>> a way to register an event loop selector somewhere and the C lib > >>>> your interfacing to provides this mechanism as well. > >>>> > >>> > >>> I see what you want to do and why now. > >>> > >>> However, I can't imagine how this could be implemented > >>> without asyncio event loop being implemented in C. > >>> Coroutines from PEP 492 aren't an indepenedant concept, > >>> they require a library/framework with an event loop > >>> to make them work. > >>> > >>> The only way to go right now is to get 'get_event_loop()' > >>> and 'loop.call_soon()' working in C using Python C API. > >>> > >>> From that point you can implement Future object in C, > >>> see 'asyncio/futures.py'; Future._schedule_callbacks, > >>> Future.set_result and Future.__iter__ methods in > >>> particular. > >>> > >>> With all that, you can easily return your future object > >>> from C methods and everything should work just fine. > >>> > >>> I know this isn't ideal, and we actually should start a > >>> separate discussion on how we can improve things on > >>> our asyncio C side. > >> > >> I'm not expecting this to be ready anytime soon, > >> just wanted to bring this up, since it may be something > >> that has to be considered in the async/await API design > >> early rather than later to not make it too complex to > >> add later on. > >> > >> I'm not sure, but going through the Python API from C > >> can potentially result in the code to reenter which sounds > >> like it could cause some unwanted race conditions. A C API > >> would avoid this. > >> > >> Thanks, > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > >> Professional Python Services directly from the Source (#1, Apr 20 2015) > >>>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ > >>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > >> ________________________________________________________________________ > >> > >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > >> > >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > >> Registered at Amtsgericht Duesseldorf: HRB 46611 > >> http://www.egenix.com/company/contact/ > >> _______________________________________________ > >> Python-ideas mailing list > >> Python-ideas at python.org > >> https://mail.python.org/mailman/listinfo/python-ideas > >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaiser.yann at gmail.com Mon Apr 20 23:39:54 2015 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Mon, 20 Apr 2015 21:39:54 +0000 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> Message-ID: Browsing the PEP and this thread, I can't help but notice there is little mention of the possibility (or rather, the lack thereof in the current version of the proposal) of writing async generators(ie. usable in async for/await for) within the newly proposed coroutines other than "yield [from] is not allowed within an async function". I think this should be given more thought, despite the technical difficulties involved (namely, new-style coroutines are still implemented as generators). While the proposal already makes it less of a burden to write asynchronous iterables through "async for", __aiter__ and __anext__, in synchronous systems writing a generator function(or having one as __iter__) is most often preferred over writing a class with __iter__ and __next__ methods. A use case I've come across using asyncio, assuming yield could be somehow allowed within a new-style coroutine: A web API provides paginated content with few filtering options. You have something that fetches those pages: async def fetch_pages(url, params): while True: page = await(request(url, params)) feed = page.decode_json() if not feed['items']: # no items when we reach the last page return yield feed It would be sorta okay to have this as a class with __aiter__ and __anext__ methods. It would be necessary anyway. Now we want an iterator for every item. We won't really want to iterate on pages anyway, we'll want to iterate on items. With yield available, this is really easy! async def fetch_items(pages): async for page in pages: yield from page['items'] Without, it becomes more difficult class ItemFetcher: def __init__(self, pages): self.pages = pages async def __aiter__(self): self.pages_itor = await self.pages.__aiter__() self.items = iter(()) return self async def __anext__(self): try: try: return next(self.page) except StopIteration: page = await self.pages_itor.__anext__() self.page = iter(page['items']) return next(self.page) except StopIteration as exc: raise StopAsyncIteration(StopIteration.value) from exc Whoa there! This is complicated and difficult to understand. But we really want to iterate on items so we leave it in anyway. Here the language failed already. It made a monster out of what could have been expressed simply. What if we only want new items? async def new_items(items): async for item in items: if is_new_item(item): yield item Without yield: class NewItems: def __init__(self, items): self.items = items async def __aiter__(self): self.itor = await self.items.__aiter__() return self async def __anext__(self): async for item in self.itor: if is_new_item(item): return item raise StopAsyncIteration This isn't as bad as the previous example, but I'll be the first to admit I'll use "if not is_new_item(item): continue" in client code instead. In bullet point form, skipping the possibility of having yield within coroutine functions causes the following: * To write async iterators, you have to use the full async-iterable class form. * Iterators written in class form can't have a for loop over their arguments, because their behavior is spread over multiple methods. * Iterators are unwieldy when used manually * Async iterators even more so => If you want to make an async iterator, it's complicated => If you want to make an async iterator that iterates over an iterator, it's more complicated => If you want to make an async iterator that iterates over an async iterator, it's even more complicated I therefore think a way to have await and yield coexist should be looked into. Solutions include adding new bytecodes or adding a flag to the YIELD_VALUE and YIELD_FROM bytecodes. -- Yann On Mon, 20 Apr 2015 at 08:43 Guido van Rossum wrote: > On Fri, Apr 17, 2015 at 5:21 PM, Greg Ewing > wrote: > >> Yury Selivanov wrote: >> >>> >>> Here's my proposal to add async/await in Python. >>> >> >> You've essentially reinvented PEP 3152 - Cofunctions. >> >> https://www.python.org/dev/peps/pep-3152/ >> > > I *thought* Yury's proposal sounded familiar! :-) > > >> Here's a summary of the relationships between them: >> >> PEP 3152 Nearest equivalent in Yury's PEP >> -------- -------------------------------- >> >> codef f(args): async def f(args): >> >> cocall f(args) await f(args) >> >> __cocall__ __await__ >> >> costart() async_def() >> >> There is currently no equivalent of "async for" and >> "async with" in PEP 3152, but they could easily be added. >> I would probably spell them "cofor" and "cowith". >> > > Unfortunately that's a lot of new reserved words, and they aren't even > words. :-) I like the elegance of Yury's idea of prefixing various > statements with "async". It also happens to easy introduction, since the > lexer can be adapted (for a few releases at least) not to treat "async" as > a reserved word unless followed by "def" etc. > > >> As the author of PEP 3152 I'm obviously biased, but I >> think my spellings are more elegant and less disruptive >> to reading of the code. >> > > As the BDFL I'm perhaps also biased (Yury went over early stages of his > draft with me at PyCon) but I prefer his spellings. > > >> PEP 3152 is currently marked as deferred. Maybe it's >> time to revive it? If Yury's pep is to be considered, >> we ought to discuss the relative merits of the two. > > > I wonder if, instead of putting forward a competing proposal that is so > similar, you could help Yury with PEP 492? I'm sure he'd happily take you > as a co-author. > > I have one question about PEP 3152. IIUC, it states that cofunctions > cannot be called except using cocall, and there seems to be syntax that > enforces that the "argument" to cocall looks like a function call. This > would rule out things that are normally always allowed in Python, if you > have > ``` > return foo(args) > ``` > you can replace that with > ``` > r = foo(args) > return r > ``` > But the way I read PEP 3152 the call has to occur syntactically in the > cocall form, so > ``` > cocall foo(args) > ``` > cannot be replaced with > ``` > x = foo(args) > cocall x > ``` > This ability is occasionally useful in asyncio, for example when creating > a task, we can write > ``` > x = Task(foo(args)) > ``` > and then someone else can write > ``` > yield from x > ``` > Your cocall syntax is lacking such ability, since cocall insist on call > syntax ("cocall x" is a syntax error). It would seem that in general your > cocall is not a substitution for yield from when used with a future or > task, and that looks like a real deficiency to me. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Tue Apr 21 00:07:29 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 20 Apr 2015 18:07:29 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> Message-ID: <553578A1.3020404@gmail.com> Hi Yann, Thanks for the feedback! Answers below. On 2015-04-20 5:39 PM, Yann Kaiser wrote: > Browsing the PEP and this thread, I can't help but notice there is little > mention of the possibility (or rather, the lack thereof in the current > version of the proposal) of writing async generators(ie. usable in async > for/await for) within the newly proposed coroutines other than "yield > [from] is not allowed within an async function". > > I think this should be given more thought, despite the technical > difficulties involved (namely, new-style coroutines are still implemented > as generators). > > While the proposal already makes it less of a burden to write asynchronous > iterables through "async for", __aiter__ and __anext__, in synchronous > systems writing a generator function(or having one as __iter__) is most > often preferred over writing a class with __iter__ and __next__ methods. > > A use case I've come across using asyncio, assuming yield could be somehow > allowed within a new-style coroutine: > > A web API provides paginated content with few filtering options. You have > something that fetches those pages: > > async def fetch_pages(url, params): > while True: > page = await(request(url, params)) > feed = page.decode_json() > if not feed['items']: # no items when we reach the last page > return > yield feed > > It would be sorta okay to have this as a class with __aiter__ and __anext__ > methods. It would be necessary anyway. > > Now we want an iterator for every item. We won't really want to iterate on > pages anyway, we'll want to iterate on items. With yield available, this is > really easy! > > async def fetch_items(pages): > async for page in pages: > yield from page['items'] > > Without, it becomes more difficult > > class ItemFetcher: > def __init__(self, pages): > self.pages = pages > > async def __aiter__(self): > self.pages_itor = await self.pages.__aiter__() > self.items = iter(()) > return self > > async def __anext__(self): > try: > try: > return next(self.page) > except StopIteration: > page = await self.pages_itor.__anext__() > self.page = iter(page['items']) > return next(self.page) > except StopIteration as exc: > raise StopAsyncIteration(StopIteration.value) from exc > > Whoa there! This is complicated and difficult to understand. But we really > want to iterate on items so we leave it in anyway. While I understand what you're trying to achieve here, the code doesn't look correct. Could you please clone the ref implementation and make it work first? > > Here the language failed already. It made a monster out of what could have > been expressed simply. > > What if we only want new items? > > async def new_items(items): > async for item in items: > if is_new_item(item): > yield item > > Without yield: > > class NewItems: > def __init__(self, items): > self.items = items > > async def __aiter__(self): > self.itor = await self.items.__aiter__() > return self > > async def __anext__(self): > async for item in self.itor: > if is_new_item(item): > return item > raise StopAsyncIteration > > This isn't as bad as the previous example, but I'll be the first to admit > I'll use "if not is_new_item(item): continue" in client code instead. > > In bullet point form, skipping the possibility of having yield within > coroutine functions causes the following: > > * To write async iterators, you have to use the full async-iterable class > form. > * Iterators written in class form can't have a for loop over their > arguments, because their behavior is spread over multiple methods. > * Iterators are unwieldy when used manually > * Async iterators even more so > => If you want to make an async iterator, it's complicated > => If you want to make an async iterator that iterates over an iterator, > it's more complicated > => If you want to make an async iterator that iterates over an async > iterator, it's even more complicated > > I therefore think a way to have await and yield coexist should be looked > into. > > Solutions include adding new bytecodes or adding a flag to the YIELD_VALUE > and YIELD_FROM bytecodes. > > -- Yann All in all, I understand your case. But also, I know that you have this case because you're trying to think in generators and how to combine them. Which is a good pattern, but unfortunately, this pattern had never worked with generator-based coroutines either. Imagine that there were no PEP 492. That you only can use asyncio and 'yield from'. You would design your code in a different way then. I spent a great deal of time thinking if it's possible to combine yields and awaits in one coroutine function. Unfortunately, even if it is possible, it will require horrible hacks and will complicate the implementation tremendously. Moreover, I think that combining 'yield' and 'yield from' expressions with 'await' will only create confusion and contradicts with the main intent of the PEP, which is to *remove that confusion*. I'd prefer to make PEP 492 maximally minimal in this regard. Since it's prohibited to use 'yield' in coroutines, we may allow it in later Python versions. (Although I'm certain, that we'll need a new keyword for 'yield' in coroutines.) At this point, PEP 492 is created *specifically* to address existing pain-points that asyncio developers have, no more. I don't want to create new programming patterns or new concepts of generator-coroutine hybrid. I think that can and should be covered in future PEPs in future Pythons. If someone smarter than me can figure out a way to do this in a non-confusing way that won't require duplicating genobject.c and fourth of ceval.c I'd be glad to update the PEP. Thanks, Yury From greg.ewing at canterbury.ac.nz Tue Apr 21 01:32:48 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 21 Apr 2015 11:32:48 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <55346E15.9040204@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> Message-ID: <55358CA0.4040406@canterbury.ac.nz> Yury Selivanov wrote: > I added a section about PEP 3152 to PEP 492: > https://hg.python.org/peps/rev/428c7c753500 > > Please take a look, and feel free to correct me if I > made any mistakes or forgot to specify something. From PEP 492: > 2. await is defined in almost the same way as yield from in the grammar (it is > later enforced that await can only be inside async def ). It is possible to > simply write await future , whereas cocall always requires parentheses. The reason for this is to make cocalls grammatically similar to ordinary calls, so that a cocall can be used in any context where an ordinary call can be used. This is not the case with yield-from, which has to be enclosed in parentheses except it certain special contexts. If I understand your grammar correctly, "await" has the same characteristics as yield-from in this regard. > 3. To make asyncio work with PEP 3152 it would be required to modify > @asyncio.coroutine decorator to wrap all functions in an object with a > __cocall__ method. That's not the only way to address this issue. I'll have to think about it further, but I don't think it would harm anything to have ordinary generator functions implement __cocall__. Then asyncio functions, and any other existing generator-based coroutines, could be used directly in cocalls without requiring any adaptation. Alternatively, a decorator equivalent to your async_def could be provided that turns an ordinary generator function into a cofunction. There would not be any additional overhead involved in calling such a function. > 4. Since it is impossible to call a cofunction without a cocall keyword, it > automatically prevents the common mistake of forgetting to use yield from on > generator-based coroutines. This proposal addresses this problem with a > different approach, see Debugging Features . This is an important part of the philosophy behind PEP 3152. With my approach, there is no additional overhead involved in enforcing it, so there is no need to relegate it to a debugging feature, with the problems that you point out. -- Greg From yselivanov.ml at gmail.com Tue Apr 21 01:59:09 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 20 Apr 2015 19:59:09 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <55358CA0.4040406@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <55358CA0.4040406@canterbury.ac.nz> Message-ID: <553592CD.9060305@gmail.com> Hi Greg, On 2015-04-20 7:32 PM, Greg Ewing wrote: > Yury Selivanov wrote: >> I added a section about PEP 3152 to PEP 492: >> https://hg.python.org/peps/rev/428c7c753500 >> >> Please take a look, and feel free to correct me if I >> made any mistakes or forgot to specify something. > > From PEP 492: > >> 2. await is defined in almost the same way as yield from in the >> grammar (it is >> later enforced that await can only be inside async def ). It is >> possible to >> simply write await future , whereas cocall always requires parentheses. > > The reason for this is to make cocalls grammatically similar > to ordinary calls, so that a cocall can be used in any context > where an ordinary call can be used. This is not the case with > yield-from, which has to be enclosed in parentheses except it > certain special contexts. If I understand your grammar correctly, > "await" has the same characteristics as yield-from in this regard. I'd say it's closer to 'yield'. So yes, you have to use parens when writing 'print((await fut))', but since I use parens and yield/yield-froms extremely rare myself and don't observe this pattern to be common in other code too, I don't think it's an issue. I think that parens here is a nice compromise. People already know how to use yields, and when to put parens around them. I don't want them to struggle with different grammar with 'await'. > >> 3. To make asyncio work with PEP 3152 it would be required to modify >> @asyncio.coroutine decorator to wrap all functions in an object with a >> __cocall__ method. > > That's not the only way to address this issue. I'll have to think > about it further, but I don't think it would harm anything to have > ordinary generator functions implement __cocall__. Then asyncio > functions, and any other existing generator-based coroutines, > could be used directly in cocalls without requiring any adaptation. > > Alternatively, a decorator equivalent to your async_def could be > provided that turns an ordinary generator function into a cofunction. > There would not be any additional overhead involved in calling such > a function. I think you're right here, we can implement __cocall__ on all generators and let it work only for generators with 'CO_ASYNC' flag (btw, I'm renaming it to 'CO_COROUTINE' for consistency). The only problem is that if we apply this flag to coroutines in asyncio, a lot of code can be broken. I.e. whenever someone passed around generator-objects, or used syntax like 'o = gen(); yield from o' there will be a *runtime* error. Moreover, in the PEP 492 we define *awaitable* as a Future-like or a *coroutine-object*. We also require __a*__ methods to return *awaitables*. Hence you can implement patterns like the following: def __aenter__(self): // a function returning awaitable if something(): return coroutine_func() else: return future() or async def __aenter__(self): // a coroutine print('hi') If we disallow calling coroutines (requiring "await <..>()" syntax) this kind of patterns won't be possible. > >> 4. Since it is impossible to call a cofunction without a cocall >> keyword, it >> automatically prevents the common mistake of forgetting to use yield >> from on >> generator-based coroutines. This proposal addresses this problem with a >> different approach, see Debugging Features . > > This is an important part of the philosophy behind PEP 3152. With > my approach, there is no additional overhead involved in enforcing > it, so there is no need to relegate it to a debugging feature, with > the problems that you point out. > Agree. I kind of like this design. See my above comments though. Thanks! Yury From greg.ewing at canterbury.ac.nz Tue Apr 21 08:06:31 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 21 Apr 2015 18:06:31 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553578A1.3020404@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <553578A1.3020404@gmail.com> Message-ID: <5535E8E7.80907@canterbury.ac.nz> Yury Selivanov wrote: > Since it's prohibited to use 'yield' in coroutines, we may > allow it in later Python versions. (Although I'm certain, that > we'll need a new keyword for 'yield' in coroutines.) The obvious way to spell it would be "async yield" (and we would probably also want "async yield from"). > If someone smarter than me can figure out a way to do this > in a non-confusing way that won't require duplicating genobject.c > and fourth of ceval.c I'd be glad to update the PEP. I don't think it would be necessary to duplicate genobject.c. The implementation would probably go something like this: * Use of 'async yield' in an async def function sets a new flag marking it as an async generator. * 'async yield x' raises StopIteration(x) so that it becomes the return value of the current __anext__ call. * Returning from an async generator raises StopAsyncIteration. I'm not sure about 'async yield from'. If I think about this any more my head will probably start to hurt, so I'll stop now. I agree that this might be better left for a later PEP. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 21 08:21:37 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 21 Apr 2015 18:21:37 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553592CD.9060305@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <55358CA0.4040406@canterbury.ac.nz> <553592CD.9060305@gmail.com> Message-ID: <5535EC71.1060900@canterbury.ac.nz> Yury Selivanov wrote: > The only problem is that if we apply this flag to coroutines > in asyncio, a lot of code can be broken. I.e. whenever someone > passed around generator-objects, or used syntax like > 'o = gen(); yield from o' there will be a *runtime* error. Hmmm, using the decorator approach, there might need to be a way of flagging a function so that both __call__ and __cocall__ work. That should be enough, since in PEP 3152 there is nothing special about the iterator returned by __cocall__. > def __aenter__(self): // a function returning awaitable > if something(): > return coroutine_func() > else: > return future() In a world where PEP 3152 were established, things like tasks and futures would implement __cocall__, and if it were to have something like __aenter__ it would be defined as a cofunction. So you would write that as codef __aenter__(self): if something(): return cocall coroutine_func() else: return cocall future() -- Greg From kaiser.yann at gmail.com Tue Apr 21 09:28:20 2015 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Tue, 21 Apr 2015 07:28:20 +0000 Subject: [Python-ideas] async/await in Python In-Reply-To: <553578A1.3020404@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <553578A1.3020404@gmail.com> Message-ID: On Tue, 21 Apr 2015 at 00:07 Yury Selivanov wrote: > Hi Yann, > > > Thanks for the feedback! Answers below. > Thanks for hearing it! > On 2015-04-20 5:39 PM, Yann Kaiser wrote: > > Browsing the PEP and this thread, I can't help but notice there is little > > mention of the possibility (or rather, the lack thereof in the current > > version of the proposal) of writing async generators(ie. usable in async > > for/await for) within the newly proposed coroutines other than "yield > > [from] is not allowed within an async function". > > > > I think this should be given more thought, despite the technical > > difficulties involved (namely, new-style coroutines are still implemented > > as generators). > > > > While the proposal already makes it less of a burden to write > asynchronous > > iterables through "async for", __aiter__ and __anext__, in synchronous > > systems writing a generator function(or having one as __iter__) is most > > often preferred over writing a class with __iter__ and __next__ methods. > > > > A use case I've come across using asyncio, assuming yield could be > somehow > > allowed within a new-style coroutine: > > > > A web API provides paginated content with few filtering options. You have > > something that fetches those pages: > > > > async def fetch_pages(url, params): > > while True: > > page = await(request(url, params)) > > feed = page.decode_json() > > if not feed['items']: # no items when we reach the last page > > return > > yield feed > > > > It would be sorta okay to have this as a class with __aiter__ and > __anext__ > > methods. It would be necessary anyway. > > > > Now we want an iterator for every item. We won't really want to iterate > on > > pages anyway, we'll want to iterate on items. With yield available, this > is > > really easy! > > > > async def fetch_items(pages): > > async for page in pages: > > yield from page['items'] > > > > Without, it becomes more difficult > > > > class ItemFetcher: > > def __init__(self, pages): > > self.pages = pages > > > > async def __aiter__(self): > > self.pages_itor = await self.pages.__aiter__() > > self.items = iter(()) > > return self > > > > async def __anext__(self): > > try: > > try: > > return next(self.page) > > except StopIteration: > > page = await self.pages_itor.__anext__() > > self.page = iter(page['items']) > > return next(self.page) > > except StopIteration as exc: > > raise StopAsyncIteration(StopIteration.value) from exc > > > > Whoa there! This is complicated and difficult to understand. But we > really > > want to iterate on items so we leave it in anyway. > > While I understand what you're trying to achieve here, the > code doesn't look correct. Could you please clone the ref > implementation and make it work first? > Hadn't realized there was one. Will do. > > > > Here the language failed already. It made a monster out of what could > have > > been expressed simply. > > > > What if we only want new items? > > > > async def new_items(items): > > async for item in items: > > if is_new_item(item): > > yield item > > > > Without yield: > > > > class NewItems: > > def __init__(self, items): > > self.items = items > > > > async def __aiter__(self): > > self.itor = await self.items.__aiter__() > > return self > > > > async def __anext__(self): > > async for item in self.itor: > > if is_new_item(item): > > return item > > raise StopAsyncIteration > > > > This isn't as bad as the previous example, but I'll be the first to admit > > I'll use "if not is_new_item(item): continue" in client code instead. > > > > In bullet point form, skipping the possibility of having yield within > > coroutine functions causes the following: > > > > * To write async iterators, you have to use the full async-iterable class > > form. > > * Iterators written in class form can't have a for loop over their > > arguments, because their behavior is spread over multiple methods. > > * Iterators are unwieldy when used manually > > * Async iterators even more so > > => If you want to make an async iterator, it's complicated > > => If you want to make an async iterator that iterates over an iterator, > > it's more complicated > > => If you want to make an async iterator that iterates over an async > > iterator, it's even more complicated > > > > I therefore think a way to have await and yield coexist should be looked > > into. > > > > Solutions include adding new bytecodes or adding a flag to the > YIELD_VALUE > > and YIELD_FROM bytecodes. > > > > -- Yann > > > All in all, I understand your case. > > But also, I know that you have this case because you're > trying to think in generators and how to combine them. Which is > a good pattern, but unfortunately, this pattern had never > worked with generator-based coroutines either. > It was, sort of. I wrote a decorator that did it using a wrapper around results to be iterated on, thus multiplexing the use of the "yield channel" between Futures and Results. Without some form of "asynchronous for loop" that is now being proposed, the looping side has to yield each value it gets from the iterable, and that yield can potentially raise an alternate StopIteration (eg. when you only know you've run out of items after making a request) Here's an example that used both looping and yielding values: @asyncio_generator def __iter__(self, Result): for f in self.iter_pages(): listing = yield f for link in listing: yield Result(link) Tangentially, this could be: async def __aiter__(self): async for listing in self.iter_pages(): for link in listing: yield link Anyway, if you want to look behind the curtains... def asyncio_generator(func): def wrapped(*args, **kwargs): class Result(object): def __init__(self, val): self.val = val gen = func(*args, Result=Result, **kwargs) return _AsyncioGeneratorIterator(gen, Result) return wrapped class LateStopIteration(RuntimeError): def __str__(self): return "Attempted to stop iteration after non-result" class _AsyncioGeneratorIterator(object): def __init__(self, gen, result_cls): self.gen = gen self.result_cls = result_cls def __iter__(self): return self def __next__(self): val = next(self.gen) return self.process_val(val) @asyncio.coroutine def process_val(self, val): while not isinstance(val, self.result_cls): try: try: data = yield from val except BaseException as exc: val = self.gen.throw(*sys.exc_info()) else: val = self.gen.send(data) except StopIteration as exc: raise LateStopIteration from exc return val.val @asyncio.coroutine def as_list(self): lis = [] for fut in self: try: val = yield from fut except LateStopIteration: break lis.append(val) return lis It looks like a hack, and that's because it is one :-) It passes a "Result" class as keyword argument to the decorated function when called and expects it to yield either coroutines or Result instance. I'm pretty sure there was a reason I couldn't just yield from those coros inside the decorated function. It's been a while... Imagine that there were no PEP 492. That you only can use > asyncio and 'yield from'. You would design your code > in a different way then. > > I spent a great deal of time thinking if it's possible to > combine yields and awaits in one coroutine function. > > Unfortunately, even if it is possible, it will require horrible > hacks and will complicate the implementation tremendously. > > Moreover, I think that combining 'yield' and 'yield from' > expressions with 'await' will only create confusion and contradicts > with the main intent of the PEP, which is to *remove that confusion*. > > I'd prefer to make PEP 492 maximally minimal in this regard. > Since it's prohibited to use 'yield' in coroutines, we may > allow it in later Python versions. (Although I'm certain, that > we'll need a new keyword for 'yield' in coroutines.) > I think all in all it could be upgraded to work with the proposal, thus eliminating the need to yield each value we get from the iterator. > At this point, PEP 492 is created *specifically* to address > existing pain-points that asyncio developers have, no more. I don't want to create new programming patterns or new concepts > of generator-coroutine hybrid. I think that can and should be > covered in future PEPs in future Pythons. > Well... I didn't make this up out of thin air :-) > If someone smarter than me can figure out a way to do this > in a non-confusing way that won't require duplicating genobject.c > and fourth of ceval.c I'd be glad to update the PEP. Well the above strategy could be imitated, multiplexing the yield channel between yields meant for the event loop and those meant for the asynchronous iteration protocol. It could be done with an additional parameter to YIELD_VALUE (I'm assuming await compiles down to it or YIELD_FROM ?), although it would be redundant for functions not using both concepts at the same time. Still, it's just one byte. YIELD_FROM... my head explosion warnings are also being set off. As Greg pointed out, the presence of yield in an async function would have it return an object also supporting __aiter__ and __anext__. Thanks, > Yury > > Yann > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Apr 21 09:42:21 2015 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 21 Apr 2015 16:42:21 +0900 Subject: [Python-ideas] async/await in Python (C API ?) In-Reply-To: References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <55346E15.9040204@gmail.com> <5534FCA4.2030909@egenix.com> <553501B0.10201@gmail.com> <553511F4.3050204@egenix.com> <55351871.3040902@gmail.com> <55352610.1070508@egenix.com> <55354984.8030503@egenix.com> Message-ID: I'm +1 for C API. ? ? Writing server or client in C To write fast networking application, low-level I/O should be done without any Python call. For example, fast HTTP server should accept, read and parse incoming request in C. To achieve it, minimum part of eventloop should provide Capsule for watching fds https://docs.python.org/3.4/library/asyncio-eventloop.html#watch-file-descriptors I think we can prototyping in third party event loop like aiouv . ? ? Accelerating asyncio application written in Python I've found bottleneck of aiohttp is stream. aiohttp has own FlowControlStreamReader. It extends aiohttp.streams.StreamReader and aiohttp.streams.StreamReader extends asyncio.StreamReader. It makes reading from stream slow. To make it faster, imports some feature from aiohttp's stream to asyncio stream, and rewrite stream in C like io module was rewritten. But this is hard job because stream provides/uses coroutines. Before rewriting asyncio.stream, Python should provide way to use and provide coroutine in C. And aiohttp may be able to use Cython to prototype faster stream. On Tue, Apr 21, 2015 at 4:39 AM, Ludovic Gasc wrote: > +1 for C Accelerators for AsyncIO. > I don't have the level to implement that, nevertheless I can contribute > freely with two other manners: benchmarks and launch that on several > daemons to track bugs. > > -- > Ludovic Gasc (GMLudo) > http://www.gmludo.eu/ > > 2015-04-20 20:46 GMT+02:00 M.-A. Lemburg : > >> On 20.04.2015 20:16, Andrew Svetlov wrote: >> > Marc-Andre, I see your need but we should write C Accelerators for >> > asyncio first. >> > The task is not related to the PEP directly, while I like to working on >> it soon. >> > The problem is: accelerator should be backward compatibility and >> > optional (no problem with it). >> > But making C API requires much more: we need to implement the most >> > critical parts of asyncio in C and then invent C API for that. >> > Anyway, thank you for inspiration for desired C Async API. >> >> Understood, and thanks for pushing all this. Looking forward to >> some future PEP which outlines the C API for async/await :-) >> >> BTW: If you need some financial backup for these things, please >> consider writing a grant application to the PSF: >> >> https://www.python.org/psf/grants/ >> >> Cheers, >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Source (#1, Apr 20 2015) >> >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >> >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ >> ________________________________________________________________________ >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> http://www.egenix.com/company/contact/ >> >> > On Mon, Apr 20, 2015 at 7:15 PM, M.-A. Lemburg wrote: >> >> On 20.04.2015 17:17, Yury Selivanov wrote: >> >>> >> >>> On 2015-04-20 10:49 AM, M.-A. Lemburg wrote: >> >>>> On 20.04.2015 15:40, Yury Selivanov wrote: >> >>>>> Hi Marc-Andre, >> >>>>> >> >>>>> On 2015-04-20 9:18 AM, M.-A. Lemburg wrote: >> >>>>>> Hi Yury, >> >>>>>> >> >>>>>> do you have any plans of also exposing an easy to use C API >> >>>>>> for async and await ? >> >>>>>> >> >>>>>> I think this would be useful to e.g. implement asynchronous >> >>>>>> database access. Most of those adapters are written in C and >> >>>>>> need so a C API would help a lot with this. >> >>>>> I think one way to have a convenient C API is to implement >> >>>>> a Future-like object in C -- an object with __await__ method >> >>>>> that should return an iterator, which should also be >> >>>>> implemented in C and provide a way to attach callback >> >>>>> or to *pipeline* C functions. >> >>>>> >> >>>>> This way C code should seamlessly integrate with asyncio. >> >>>>> >> >>>>> I doubt that we have time to include this kind of API in >> >>>>> the PEP, but it can be developed as a separate library. >> >>>>> >> >>>>> Your thoughts? >> >>>> I'm just sketching here, since I don't have much experience >> >>>> with writing async code (I'm using threads and multiple >> >>>> processes instead) and it probably shows ;-) >> >>>> >> >>>> From the C side of things, I think using callbacks >> >>>> and a way to register/unregister polling functions >> >>>> would be great to have. >> >>>> >> >>>> The C code could then run like this in polling mode: >> >>>> >> >>>> long_running_query(...) { >> >>>> /* Start long running operation... creating a handle for it */ >> >>>> PyAsync_RegisterPoll(mypolling_fct, handle, polling_interval); >> >>>> PyAsync_ReturnAwait(); >> >>>> } >> >>>> >> >>>> mypolling_fct(int handle) { >> >>>> /* Check completion of operation */ >> >>>> if (completed) { >> >>>> /* Fetch results into value */ >> >>>> PyAsync_UnregisterPoll(mypolling_fct, handle); >> >>>> PyAsync_ReturnReady(value); >> >>>> } >> >>>> else >> >>>> PyAsync_ReturnContinue(); >> >>>> } >> >>>> >> >>>> and like this in callback mode: >> >>>> >> >>>> long_running_query(...) { >> >>>> /* Create a handle for a long running operation... */ >> >>>> PyAsync_RegisterCallback(myquery_done, handle, timeout); >> >>>> /* Start long running operation... */ >> >>>> PyAsync_ReturnAwait(); >> >>>> } >> >>>> >> >>>> myquery_done(int handle, PyObject *value) { >> >>>> PyAsync_UnregisterCallback(myquery_done, handle); >> >>>> PyAsync_ReturnReady(value); >> >>>> } >> >>>> >> >>>> Both mechanisms will likely have to move the long running >> >>>> operation into a separate (blocking) thread, unless there's also >> >>>> a way to register an event loop selector somewhere and the C lib >> >>>> your interfacing to provides this mechanism as well. >> >>>> >> >>> >> >>> I see what you want to do and why now. >> >>> >> >>> However, I can't imagine how this could be implemented >> >>> without asyncio event loop being implemented in C. >> >>> Coroutines from PEP 492 aren't an indepenedant concept, >> >>> they require a library/framework with an event loop >> >>> to make them work. >> >>> >> >>> The only way to go right now is to get 'get_event_loop()' >> >>> and 'loop.call_soon()' working in C using Python C API. >> >>> >> >>> From that point you can implement Future object in C, >> >>> see 'asyncio/futures.py'; Future._schedule_callbacks, >> >>> Future.set_result and Future.__iter__ methods in >> >>> particular. >> >>> >> >>> With all that, you can easily return your future object >> >>> from C methods and everything should work just fine. >> >>> >> >>> I know this isn't ideal, and we actually should start a >> >>> separate discussion on how we can improve things on >> >>> our asyncio C side. >> >> >> >> I'm not expecting this to be ready anytime soon, >> >> just wanted to bring this up, since it may be something >> >> that has to be considered in the async/await API design >> >> early rather than later to not make it too complex to >> >> add later on. >> >> >> >> I'm not sure, but going through the Python API from C >> >> can potentially result in the code to reenter which sounds >> >> like it could cause some unwanted race conditions. A C API >> >> would avoid this. >> >> >> >> Thanks, >> >> -- >> >> Marc-Andre Lemburg >> >> eGenix.com >> >> >> >> Professional Python Services directly from the Source (#1, Apr 20 >> 2015) >> >>>>> Python Projects, Coaching and Consulting ... >> http://www.egenix.com/ >> >>>>> mxODBC Plone/Zope Database Adapter ... >> http://zope.egenix.com/ >> >>>>> mxODBC, mxDateTime, mxTextTools ... >> http://python.egenix.com/ >> >> >> ________________________________________________________________________ >> >> >> >> ::::: Try our mxODBC.Connect Python Database Interface for free ! >> :::::: >> >> >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> >> http://www.egenix.com/company/contact/ >> >> _______________________________________________ >> >> Python-ideas mailing list >> >> Python-ideas at python.org >> >> https://mail.python.org/mailman/listinfo/python-ideas >> >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > >> > >> > >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Tue Apr 21 18:13:40 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 21 Apr 2015 12:13:40 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <553578A1.3020404@gmail.com> Message-ID: <55367734.8000201@gmail.com> Yann, On 2015-04-21 3:28 AM, Yann Kaiser wrote: > On Tue, 21 Apr 2015 at 00:07 Yury Selivanov wrote: [...] >> If someone smarter than me can figure out a way to do this >> in a non-confusing way that won't require duplicating genobject.c >> and fourth of ceval.c I'd be glad to update the PEP. > > Well the above strategy could be imitated, multiplexing the yield channel > between yields meant for the event loop and those meant for the > asynchronous iteration protocol. It could be done with an additional > parameter to YIELD_VALUE (I'm assuming await compiles down to it or > YIELD_FROM ?), although it would be redundant for functions not using both > concepts at the same time. Still, it's just one byte. > YIELD_FROM... my head explosion warnings are also being set off. > > As Greg pointed out, the presence of yield in an async function would have > it return an object also supporting __aiter__ and __anext__. > Yann, I think that there are no fundamental reasons preventing us from having a coroutine-generator hybrid. The semantics will probably be like that: def foo(): # generator function yield 1 foo() # generator object async def bar(): # coroutine-generator function async yield 1 # or just 'yield 1' bar() # coroutine-generator object Maybe it's possible to heavily hack genobject.c and ceval.c (YIELD & YIELD_FROM implementations) to support this. But I think that the amount of hacks will be so huge, that we'll need a pair of new opcodes + new asyncgenobject.c, to at least avoid performance penalties on regular generators. Anyways, I think it's outside of the scope of PEP 492. I think that the best thing we should do now, is to make sure that we *can* have coroutine-generators in the future. I think that PEP 492, in its current state leaves a room for that. If you want to continue the discussion on this topic, I suggest you to clone the reference implementation and implement your concept. But there is no guarantee that it will land to PEP 492 though. Thanks, Yury From yselivanov.ml at gmail.com Tue Apr 21 18:20:24 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 21 Apr 2015 12:20:24 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <5535E8E7.80907@canterbury.ac.nz> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <553578A1.3020404@gmail.com> <5535E8E7.80907@canterbury.ac.nz> Message-ID: <553678C8.3000005@gmail.com> Hi Greg, On 2015-04-21 2:06 AM, Greg Ewing wrote: > Yury Selivanov wrote: >> Since it's prohibited to use 'yield' in coroutines, we may >> allow it in later Python versions. (Although I'm certain, that >> we'll need a new keyword for 'yield' in coroutines.) > > The obvious way to spell it would be "async yield" (and > we would probably also want "async yield from"). Agree! Although it looks a little bit strange, I think we will need to clearly indicate that this is not an ordinary yield expression. > >> If someone smarter than me can figure out a way to do this >> in a non-confusing way that won't require duplicating genobject.c >> and fourth of ceval.c I'd be glad to update the PEP. > > I don't think it would be necessary to duplicate genobject.c. > The implementation would probably go something like this: > > * Use of 'async yield' in an async def function sets a new > flag marking it as an async generator. > > * 'async yield x' raises StopIteration(x) so that it becomes > the return value of the current __anext__ call. > > * Returning from an async generator raises StopAsyncIteration. The semantics you describe looks correct. Although I highly doubt that it's possible to fit all this magic into existing generators implementation (it should be possible, but make it more complicated, harder to support and maybe a bit slower). > > I'm not sure about 'async yield from'. If I think about this > any more my head will probably start to hurt, so I'll > stop now. :) > > I agree that this might be better left for a later PEP. > I think that's what we should do. I'd be happy to help with that later PEP. BTW, there is one more thing about requiring a keyword to make a coroutine call. If we decide to implement coroutine-generators one day, a simple call to coroutine-generator should return a coroutine-generator object: async def foo(): async yield 1 foo() # <- coroutine-generator object In the other hand, if you don't have an 'async yield' expression, then foo() would raise an error. I'm not sure I like this duality. Thanks, Yury From yselivanov.ml at gmail.com Tue Apr 21 18:59:29 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 21 Apr 2015 12:59:29 -0400 Subject: [Python-ideas] async/await in Python In-Reply-To: <553678C8.3000005@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <553578A1.3020404@gmail.com> <5535E8E7.80907@canterbury.ac.nz> <553678C8.3000005@gmail.com> Message-ID: <553681F1.2010309@gmail.com> On 2015-04-21 12:20 PM, Yury Selivanov wrote: > BTW, there is one more thing about requiring a keyword > to make a coroutine call. If we decide to implement > coroutine-generators one day, a simple call to > coroutine-generator should return a coroutine-generator > object: > > async def foo(): > async yield 1 > > foo() # <- coroutine-generator object > > In the other hand, if you don't have an 'async yield' > expression, then > > foo() > > would raise an error. I'm not sure I like this duality. Of course we can have __cocall__ (and no __call__) for regular coroutines and __call__ (no __cocall__) for coroutine-generators, but this also looks suspicious. Yury From yselivanov.ml at gmail.com Tue Apr 21 19:27:57 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 21 Apr 2015 13:27:57 -0400 Subject: [Python-ideas] PEP 492 update Message-ID: <5536889D.7000202@gmail.com> Hi python-ideas, I'm moving the discussion of updated PEP 492 to python-dev. Thanks, Yury From ericsnowcurrently at gmail.com Wed Apr 22 01:48:47 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 21 Apr 2015 17:48:47 -0600 Subject: [Python-ideas] concurrency-local namespaces Message-ID: threading.local provides thread-local namespaces. As far as I can tell, there isn't an equivalent for coroutines (asyncio), even though I would expect they would provide the same benefits. I'd like to see coroutine-local namespaces added and would be happy to work on the problem. I plan on working on a feature that will rely on applying a thread-local context and realized that coroutines would need a similar treatment. Also, there are probably a few spots in the stdlib that would benefit (e.g. decimal contexts). Perhaps I'm missing something. Is the concurrency story with coroutines different enough that coroutine-local namespaces do not make sense? If they still make sense, is there a mechanism to uniquely identify the "currently running" coroutine like you can with threads? That would be necessary to do a naive port of thread-local namespaces. -eric From guido at python.org Wed Apr 22 01:59:32 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Apr 2015 16:59:32 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: There was some discussion of this in https://github.com/python/asyncio/issues/165 back in the days. IIRC the conclusion was that you could roll this yourself using the Tasks.current_task() class method. On Tue, Apr 21, 2015 at 4:48 PM, Eric Snow wrote: > threading.local provides thread-local namespaces. As far as I can > tell, there isn't an equivalent for coroutines (asyncio), even though > I would expect they would provide the same benefits. I'd like to see > coroutine-local namespaces added and would be happy to work on the > problem. I plan on working on a feature that will rely on applying a > thread-local context and realized that coroutines would need a similar > treatment. Also, there are probably a few spots in the stdlib that > would benefit (e.g. decimal contexts). > > Perhaps I'm missing something. Is the concurrency story with > coroutines different enough that coroutine-local namespaces do not > make sense? > > If they still make sense, is there a mechanism to uniquely identify > the "currently running" coroutine like you can with threads? That > would be necessary to do a naive port of thread-local namespaces. > > -eric > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Apr 22 02:07:38 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 21 Apr 2015 18:07:38 -0600 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: On Tue, Apr 21, 2015 at 5:59 PM, Guido van Rossum wrote: > There was some discussion of this in > https://github.com/python/asyncio/issues/165 back in the days. IIRC the > conclusion was that you could roll this yourself using the > Tasks.current_task() class method. Perfect. Thanks. I may still look into writing a "context-local" namespace type based on the one in Lib/_threading_local.py. -eric From guido at python.org Wed Apr 22 02:37:43 2015 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Apr 2015 17:37:43 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: When you do, watch out for some edge cases that may be different from the way threads work. In particular, I can't remember whether it may be possible that the current task before and after `yield from` differs, in case you have multiple event loops. At least the code that manages the data structure used by current_task() allows this possibility. On Tue, Apr 21, 2015 at 5:07 PM, Eric Snow wrote: > On Tue, Apr 21, 2015 at 5:59 PM, Guido van Rossum > wrote: > > There was some discussion of this in > > https://github.com/python/asyncio/issues/165 back in the days. IIRC the > > conclusion was that you could roll this yourself using the > > Tasks.current_task() class method. > > Perfect. Thanks. I may still look into writing a "context-local" > namespace type based on the one in Lib/_threading_local.py. > > -eric > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Apr 22 04:07:48 2015 From: greg.ewing at canterbury.ac.nz (Greg) Date: Wed, 22 Apr 2015 14:07:48 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553678C8.3000005@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <553578A1.3020404@gmail.com> <5535E8E7.80907@canterbury.ac.nz> <553678C8.3000005@gmail.com> Message-ID: <55370274.50209@canterbury.ac.nz> On 22/04/2015 4:20 a.m., Yury Selivanov wrote: > If we decide to implement > coroutine-generators one day, a simple call to > coroutine-generator should return a coroutine-generator > object: > > async def foo(): > async yield 1 > > foo() # <- coroutine-generator object > > In the other hand, if you don't have an 'async yield' > expression, then > > foo() > > would raise an error. I'm not sure I like this duality. If you don't like it, you could always require the call to the async-generator to be an "await" call, even though it can't actually suspend at that point. -- Greg From njs at pobox.com Wed Apr 22 07:36:34 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 21 Apr 2015 22:36:34 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: On Tue, Apr 21, 2015 at 4:48 PM, Eric Snow wrote: > threading.local provides thread-local namespaces. As far as I can > tell, there isn't an equivalent for coroutines (asyncio), even though > I would expect they would provide the same benefits. I'd like to see > coroutine-local namespaces added and would be happy to work on the > problem. I plan on working on a feature that will rely on applying a > thread-local context and realized that coroutines would need a similar > treatment. Also, there are probably a few spots in the stdlib that > would benefit (e.g. decimal contexts). Possibly this is off-topic for this thread but -- it's a good point that, currently, when implementing a global state variable (like the decimal context), one can bind it to a single thread by storing the state in TLS, but AFAIK there's no way to do this with coroutines. It's somewhat unpleasant that with decimal.localcontext() as ctx: ctx.prec = 10 values = (yield from db.fetchstuff()) morecodehere() can cause arbitrary other code to run with the given decimal context in effect, while also failing to guarantee that morecodehere() will run with any particular decimal context. I hoped for a second that "async with" might help with this, but checking the PEP it doesn't -- to do that it would need support context manager callbacks that were called whenever the coroutine got suspended/resumed while the 'with' block was in effect, but that's not what it does, it just allows the __enter__ and __exit__ methods themselves to be coroutines. And AFAICT the thread Guido linked (https://github.com/python/asyncio/issues/165) only considers cases where an async framework (e.g. a web server framework) is the one who wants to set global variables that can be accessed from within a set of coroutines. This works out, b/c when the same code controls both the spawning of coroutines (by creating a new Task) and setting the variables, so it can bind them together. But that doesn't seem to help for something like the decimal module context, or numpy's error handling context (http://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html). Obviously one can just document that users should always bracket any yield/await calls with explicit state setting/restore functions, but is there any better solution? -n -- Nathaniel J. Smith -- http://vorpus.org From greg.ewing at canterbury.ac.nz Wed Apr 22 07:50:39 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 22 Apr 2015 17:50:39 +1200 Subject: [Python-ideas] async/await in Python In-Reply-To: <553681F1.2010309@gmail.com> References: <553157F2.8060408@gmail.com> <5531A390.8060405@canterbury.ac.nz> <553578A1.3020404@gmail.com> <5535E8E7.80907@canterbury.ac.nz> <553678C8.3000005@gmail.com> <553681F1.2010309@gmail.com> Message-ID: <553736AF.7030407@canterbury.ac.nz> Yury Selivanov wrote: > Of course we can have __cocall__ (and no __call__) for > regular coroutines and __call__ (no __cocall__) for > coroutine-generators, but this also looks suspicious. I think it makes sense. The PEP 3152 equivalent of __aiter__ would be called __coiter__ and called with a cocall. Or maybe not. My head is starting to feel explodey again. -- Greg From guido at python.org Wed Apr 22 19:46:32 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Apr 2015 10:46:32 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds like it would be super great if the async with statement would (optionally?) notify the context manager of switching to a different task and back. On Tue, Apr 21, 2015 at 10:36 PM, Nathaniel Smith wrote: > On Tue, Apr 21, 2015 at 4:48 PM, Eric Snow > wrote: > > threading.local provides thread-local namespaces. As far as I can > > tell, there isn't an equivalent for coroutines (asyncio), even though > > I would expect they would provide the same benefits. I'd like to see > > coroutine-local namespaces added and would be happy to work on the > > problem. I plan on working on a feature that will rely on applying a > > thread-local context and realized that coroutines would need a similar > > treatment. Also, there are probably a few spots in the stdlib that > > would benefit (e.g. decimal contexts). > > Possibly this is off-topic for this thread but -- it's a good point > that, currently, when implementing a global state variable (like the > decimal context), one can bind it to a single thread by storing the > state in TLS, but AFAIK there's no way to do this with coroutines. > It's somewhat unpleasant that > > with decimal.localcontext() as ctx: > ctx.prec = 10 > values = (yield from db.fetchstuff()) > morecodehere() > > can cause arbitrary other code to run with the given decimal context > in effect, while also failing to guarantee that morecodehere() will > run with any particular decimal context. > > I hoped for a second that "async with" might help with this, but > checking the PEP it doesn't -- to do that it would need support > context manager callbacks that were called whenever the coroutine got > suspended/resumed while the 'with' block was in effect, but that's not > what it does, it just allows the __enter__ and __exit__ methods > themselves to be coroutines. > > And AFAICT the thread Guido linked > (https://github.com/python/asyncio/issues/165) only considers cases > where an async framework (e.g. a web server framework) is the one who > wants to set global variables that can be accessed from within a set > of coroutines. This works out, b/c when the same code controls both > the spawning of coroutines (by creating a new Task) and setting the > variables, so it can bind them together. But that doesn't seem to help > for something like the decimal module context, or numpy's error > handling context > (http://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html). > > Obviously one can just document that users should always bracket any > yield/await calls with explicit state setting/restore functions, but > is there any better solution? > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Apr 22 20:12:53 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 22 Apr 2015 12:12:53 -0600 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: On Wed, Apr 22, 2015 at 11:46 AM, Guido van Rossum wrote: > Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds like > it would be super great if the async with statement would (optionally?) > notify the context manager of switching to a different task and back. Interesting idea. Keep in mind that decimal.localcontext() makes use of thread-local namespaces. So we may still be better off if we can adapt that concept to coroutines. If I've understood correctly, Tasks.current_task() should give us what we need. I'm just not clear on a couple things. Where does the loop comes from? Can more than one loop be running at the same time? I probably need to go dig into the asyncio docs. -eric From guido at python.org Wed Apr 22 20:22:46 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Apr 2015 11:22:46 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: On Wed, Apr 22, 2015 at 11:12 AM, Eric Snow wrote: > On Wed, Apr 22, 2015 at 11:46 AM, Guido van Rossum > wrote: > > Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds > like > > it would be super great if the async with statement would (optionally?) > > notify the context manager of switching to a different task and back. > > Interesting idea. Keep in mind that decimal.localcontext() makes use > of thread-local namespaces. So we may still be better off if we can > adapt that concept to coroutines. If I've understood correctly, > Tasks.current_task() should give us what we need. I'm just not clear > on a couple things. Where does the loop comes from? Can more than > one loop be running at the same time? I probably need to go dig into > the asyncio docs. > Usually there's only one loop, and you get it using asyncio.get_event_loop(). A loop is tied to a thread, so that function gets the current thread's loop. You could have another loop running in another thread, but it requires an explicit choice to set that up (by default you only get a loop for the main thread). Each loop will manages a completely distinct set of tasks (and it would require some dedication and ill will to mix up tasks between loops). You'd probably need to either have a separate asyncio-aware context for decimal, or you'd have to add asyncio-awareness to the existing context class. Both seem reasonable options; the latter seems a bit more user-friendly since users can convert existing thread-based code to coroutines without having to worry about decimal context managers across I/O waits. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Apr 22 20:37:14 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 22 Apr 2015 14:37:14 -0400 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: <5537EA5A.9000701@gmail.com> Guido, On 2015-04-22 1:46 PM, Guido van Rossum wrote: > Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds like > it would be super great if the async with statement would (optionally?) > notify the context manager of switching to a different task and back. I'm not sure if it's possible to implement. PEP492 coroutines know nothing about the loop that schedules them. One way to solve this problem of TLS-like storage for coroutines in asyncio is to add a notion of 'context' to the loop. In this code review: https://codereview.appspot.com/87940044/#ps1 please take a look at patch set 1, where I add 'loop.get_context_id()' method. I also implemented a PoC of local storage based on that patch (I can't remember where is it, but it was the reason of the code review :) I'll implement it again if we resurrect this idea). With context being implemented on the loop level (with context_id passed along with callbacks) it allowed to implement local storage concepts on a lower level than Task object, solving some of the problems Nathaniel raised. All in all, I don't think that to implement local storage we need to do something to `async with`, but rather we should implement this feature in asyncio and have the local storage object shipped with it. Thanks, Yury > > On Tue, Apr 21, 2015 at 10:36 PM, Nathaniel Smith wrote: > >> On Tue, Apr 21, 2015 at 4:48 PM, Eric Snow >> wrote: >>> threading.local provides thread-local namespaces. As far as I can >>> tell, there isn't an equivalent for coroutines (asyncio), even though >>> I would expect they would provide the same benefits. I'd like to see >>> coroutine-local namespaces added and would be happy to work on the >>> problem. I plan on working on a feature that will rely on applying a >>> thread-local context and realized that coroutines would need a similar >>> treatment. Also, there are probably a few spots in the stdlib that >>> would benefit (e.g. decimal contexts). >> Possibly this is off-topic for this thread but -- it's a good point >> that, currently, when implementing a global state variable (like the >> decimal context), one can bind it to a single thread by storing the >> state in TLS, but AFAIK there's no way to do this with coroutines. >> It's somewhat unpleasant that >> >> with decimal.localcontext() as ctx: >> ctx.prec = 10 >> values = (yield from db.fetchstuff()) >> morecodehere() >> >> can cause arbitrary other code to run with the given decimal context >> in effect, while also failing to guarantee that morecodehere() will >> run with any particular decimal context. >> >> I hoped for a second that "async with" might help with this, but >> checking the PEP it doesn't -- to do that it would need support >> context manager callbacks that were called whenever the coroutine got >> suspended/resumed while the 'with' block was in effect, but that's not >> what it does, it just allows the __enter__ and __exit__ methods >> themselves to be coroutines. >> >> And AFAICT the thread Guido linked >> (https://github.com/python/asyncio/issues/165) only considers cases >> where an async framework (e.g. a web server framework) is the one who >> wants to set global variables that can be accessed from within a set >> of coroutines. This works out, b/c when the same code controls both >> the spawning of coroutines (by creating a new Task) and setting the >> variables, so it can bind them together. But that doesn't seem to help >> for something like the decimal module context, or numpy's error >> handling context >> (http://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html). >> >> Obviously one can just document that users should always bracket any >> yield/await calls with explicit state setting/restore functions, but >> is there any better solution? >> >> -n >> >> -- >> Nathaniel J. Smith -- http://vorpus.org >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Wed Apr 22 20:45:25 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 22 Apr 2015 11:45:25 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: <5537EA5A.9000701@gmail.com> References: <5537EA5A.9000701@gmail.com> Message-ID: I'll have to leave this up to you and Nathaniel and others to ponder, I've run out of time to think about more issues. Sorry! (I do think that the conclusion of that patch was to go with current_task() intead.) On Wed, Apr 22, 2015 at 11:37 AM, Yury Selivanov wrote: > Guido, > > On 2015-04-22 1:46 PM, Guido van Rossum wrote: > >> Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds like >> it would be super great if the async with statement would (optionally?) >> notify the context manager of switching to a different task and back. >> > I'm not sure if it's possible to implement. PEP492 coroutines > know nothing about the loop that schedules them. > > One way to solve this problem of TLS-like storage for coroutines > in asyncio is to add a notion of 'context' to the loop. > > In this code review: https://codereview.appspot.com/87940044/#ps1 > please take a look at patch set 1, where I add > 'loop.get_context_id()' method. > > I also implemented a PoC of local storage based on that patch > (I can't remember where is it, but it was the reason of the code > review :) I'll implement it again if we resurrect this idea). > > With context being implemented on the loop level (with context_id > passed along with callbacks) it allowed to implement local storage > concepts on a lower level than Task object, solving some of > the problems Nathaniel raised. > > All in all, I don't think that to implement local storage we > need to do something to `async with`, but rather we should > implement this feature in asyncio and have the local storage > object shipped with it. > > Thanks, > Yury > > > >> On Tue, Apr 21, 2015 at 10:36 PM, Nathaniel Smith wrote: >> >> On Tue, Apr 21, 2015 at 4:48 PM, Eric Snow >>> wrote: >>> >>>> threading.local provides thread-local namespaces. As far as I can >>>> tell, there isn't an equivalent for coroutines (asyncio), even though >>>> I would expect they would provide the same benefits. I'd like to see >>>> coroutine-local namespaces added and would be happy to work on the >>>> problem. I plan on working on a feature that will rely on applying a >>>> thread-local context and realized that coroutines would need a similar >>>> treatment. Also, there are probably a few spots in the stdlib that >>>> would benefit (e.g. decimal contexts). >>>> >>> Possibly this is off-topic for this thread but -- it's a good point >>> that, currently, when implementing a global state variable (like the >>> decimal context), one can bind it to a single thread by storing the >>> state in TLS, but AFAIK there's no way to do this with coroutines. >>> It's somewhat unpleasant that >>> >>> with decimal.localcontext() as ctx: >>> ctx.prec = 10 >>> values = (yield from db.fetchstuff()) >>> morecodehere() >>> >>> can cause arbitrary other code to run with the given decimal context >>> in effect, while also failing to guarantee that morecodehere() will >>> run with any particular decimal context. >>> >>> I hoped for a second that "async with" might help with this, but >>> checking the PEP it doesn't -- to do that it would need support >>> context manager callbacks that were called whenever the coroutine got >>> suspended/resumed while the 'with' block was in effect, but that's not >>> what it does, it just allows the __enter__ and __exit__ methods >>> themselves to be coroutines. >>> >>> And AFAICT the thread Guido linked >>> (https://github.com/python/asyncio/issues/165) only considers cases >>> where an async framework (e.g. a web server framework) is the one who >>> wants to set global variables that can be accessed from within a set >>> of coroutines. This works out, b/c when the same code controls both >>> the spawning of coroutines (by creating a new Task) and setting the >>> variables, so it can bind them together. But that doesn't seem to help >>> for something like the decimal module context, or numpy's error >>> handling context >>> (http://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html >>> ). >>> >>> Obviously one can just document that users should always bracket any >>> yield/await calls with explicit state setting/restore functions, but >>> is there any better solution? >>> >>> -n >>> >>> -- >>> Nathaniel J. Smith -- http://vorpus.org >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Apr 22 20:50:27 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 22 Apr 2015 11:50:27 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: Message-ID: On Apr 22, 2015, at 11:22, Guido van Rossum wrote: > >> On Wed, Apr 22, 2015 at 11:12 AM, Eric Snow wrote: >> On Wed, Apr 22, 2015 at 11:46 AM, Guido van Rossum wrote: >> > Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds like >> > it would be super great if the async with statement would (optionally?) >> > notify the context manager of switching to a different task and back. >> >> Interesting idea. Keep in mind that decimal.localcontext() makes use >> of thread-local namespaces. So we may still be better off if we can >> adapt that concept to coroutines. If I've understood correctly, >> Tasks.current_task() should give us what we need. I'm just not clear >> on a couple things. Where does the loop comes from? Can more than >> one loop be running at the same time? I probably need to go dig into >> the asyncio docs. > > Usually there's only one loop, and you get it using asyncio.get_event_loop(). A loop is tied to a thread, so that function gets the current thread's loop. You could have another loop running in another thread, but it requires an explicit choice to set that up (by default you only get a loop for the main thread). If we had loop-local variables as well as thread-local, and loops are (if there's more than one) themselves thread-local, does that mean we could have a wrapper to get "local storage", which returns the loop-local storage if get_event_loop returns a loop, and thread-local storage otherwise, and change decimal contexts (and maybe other things?) to use that wrapper? Or is it not as simple as that? (As a side note, since the "block"/"macro" thread last month turned into a sort of proposal for dynamically-scoped variables before fizzling out: coroutine-chain-local-storage is the reason Scheme considered adding defvar, I think in R5. It might be worth looking into why they thought the feature was unnecessary; presumably it's because there's a better way to do the same thing, and that better way might make sense for Python as well?) > Each loop will manages a completely distinct set of tasks (and it would require some dedication and ill will to mix up tasks between loops). > > You'd probably need to either have a separate asyncio-aware context for decimal, or you'd have to add asyncio-awareness to the existing context class. Both seem reasonable options; the latter seems a bit more user-friendly since users can convert existing thread-based code to coroutines without having to worry about decimal context managers across I/O waits. > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Apr 22 20:51:27 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 22 Apr 2015 14:51:27 -0400 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: References: <5537EA5A.9000701@gmail.com> Message-ID: <5537EDAF.2030300@gmail.com> Guido, Nathaniel, On 2015-04-22 2:45 PM, Guido van Rossum wrote: > I'll have to leave this up to you and Nathaniel and others to ponder, I've > run out of time to think about more issues. Sorry! (I do think that the > conclusion of that patch was to go with current_task() intead.) Yes, that was the conclusion. Because I was interested to add this functionality for web frameworks, where, as Nathaniel mentioned, you control the top-level Task object. So I gave up on pushing the 'context' idea. If we want to add a generic thread-local-like object to coroutines, I think that it has to be done on an asyncio event loop level. Because event loop is not just about coroutines, it's also about callbacks, and it is useful to *avoid* loosing context in them. Nathaniel, I'd be glad if you could glance over my old proposal. It's based on ideas I expressed in this thread: https://groups.google.com/forum/#!topic/python-tulip/zix5HQxtElg Thanks, Yury > > On Wed, Apr 22, 2015 at 11:37 AM, Yury Selivanov > wrote: > >> Guido, >> >> On 2015-04-22 1:46 PM, Guido van Rossum wrote: >> >>> Hey Nathaniel, can you bring this up in the PEP 492 thread? It sounds like >>> it would be super great if the async with statement would (optionally?) >>> notify the context manager of switching to a different task and back. >>> >> I'm not sure if it's possible to implement. PEP492 coroutines >> know nothing about the loop that schedules them. >> >> One way to solve this problem of TLS-like storage for coroutines >> in asyncio is to add a notion of 'context' to the loop. >> >> In this code review: https://codereview.appspot.com/87940044/#ps1 >> please take a look at patch set 1, where I add >> 'loop.get_context_id()' method. >> >> I also implemented a PoC of local storage based on that patch >> (I can't remember where is it, but it was the reason of the code >> review :) I'll implement it again if we resurrect this idea). >> >> With context being implemented on the loop level (with context_id >> passed along with callbacks) it allowed to implement local storage >> concepts on a lower level than Task object, solving some of >> the problems Nathaniel raised. >> >> All in all, I don't think that to implement local storage we >> need to do something to `async with`, but rather we should >> implement this feature in asyncio and have the local storage >> object shipped with it. >> >> Thanks, >> Yury >> >> >> >>> On Tue, Apr 21, 2015 at 10:36 PM, Nathaniel Smith wrote: >>> >>> On Tue, Apr 21, 2015 at 4:48 PM, Eric Snow >>>> wrote: >>>> >>>>> threading.local provides thread-local namespaces. As far as I can >>>>> tell, there isn't an equivalent for coroutines (asyncio), even though >>>>> I would expect they would provide the same benefits. I'd like to see >>>>> coroutine-local namespaces added and would be happy to work on the >>>>> problem. I plan on working on a feature that will rely on applying a >>>>> thread-local context and realized that coroutines would need a similar >>>>> treatment. Also, there are probably a few spots in the stdlib that >>>>> would benefit (e.g. decimal contexts). >>>>> >>>> Possibly this is off-topic for this thread but -- it's a good point >>>> that, currently, when implementing a global state variable (like the >>>> decimal context), one can bind it to a single thread by storing the >>>> state in TLS, but AFAIK there's no way to do this with coroutines. >>>> It's somewhat unpleasant that >>>> >>>> with decimal.localcontext() as ctx: >>>> ctx.prec = 10 >>>> values = (yield from db.fetchstuff()) >>>> morecodehere() >>>> >>>> can cause arbitrary other code to run with the given decimal context >>>> in effect, while also failing to guarantee that morecodehere() will >>>> run with any particular decimal context. >>>> >>>> I hoped for a second that "async with" might help with this, but >>>> checking the PEP it doesn't -- to do that it would need support >>>> context manager callbacks that were called whenever the coroutine got >>>> suspended/resumed while the 'with' block was in effect, but that's not >>>> what it does, it just allows the __enter__ and __exit__ methods >>>> themselves to be coroutines. >>>> >>>> And AFAICT the thread Guido linked >>>> (https://github.com/python/asyncio/issues/165) only considers cases >>>> where an async framework (e.g. a web server framework) is the one who >>>> wants to set global variables that can be accessed from within a set >>>> of coroutines. This works out, b/c when the same code controls both >>>> the spawning of coroutines (by creating a new Task) and setting the >>>> variables, so it can bind them together. But that doesn't seem to help >>>> for something like the decimal module context, or numpy's error >>>> handling context >>>> (http://docs.scipy.org/doc/numpy/reference/generated/numpy.errstate.html >>>> ). >>>> >>>> Obviously one can just document that users should always bracket any >>>> yield/await calls with explicit state setting/restore functions, but >>>> is there any better solution? >>>> >>>> -n >>>> >>>> -- >>>> Nathaniel J. Smith -- http://vorpus.org >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > From rymg19 at gmail.com Thu Apr 23 20:52:29 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 23 Apr 2015 13:52:29 -0500 Subject: [Python-ideas] Issue with sys.exit and a long Message-ID: Observe: ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1)' ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1L)' 1 ryan at DevPC-LX:~/bfort$ If I call sys.exit with a long value under Python 2, it prints it. I count this as a bug because: 1. int and long are supposed to be as similar as possible. 2. When you're using Hy (which implicitly makes all ints longs), this sucks. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Apr 23 21:00:49 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Apr 2015 12:00:49 -0700 Subject: [Python-ideas] Issue with sys.exit and a long In-Reply-To: References: Message-ID: The bug tracker would be a good place for this. :-) On Thu, Apr 23, 2015 at 11:52 AM, Ryan Gonzalez wrote: > Observe: > > ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1)' > ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1L)' > 1 > ryan at DevPC-LX:~/bfort$ > > If I call sys.exit with a long value under Python 2, it prints it. I count > this as a bug because: > > 1. int and long are supposed to be as similar as possible. > 2. When you're using Hy (which implicitly makes all ints longs), this > sucks. > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Thu Apr 23 21:38:52 2015 From: nad at acm.org (Ned Deily) Date: Thu, 23 Apr 2015 12:38:52 -0700 Subject: [Python-ideas] Issue with sys.exit and a long References: Message-ID: In article , Guido van Rossum wrote: > The bug tracker would be a good place for this. :-) > On Thu, Apr 23, 2015 at 11:52 AM, Ryan Gonzalez > wrote: > > > Observe: > > > > ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1)' > > ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1L)' > > 1 > > ryan at DevPC-LX:~/bfort$ > > > > If I call sys.exit with a long value under Python 2, it prints it. I count > > this as a bug because: > > > > 1. int and long are supposed to be as similar as possible. > > 2. When you're using Hy (which implicitly makes all ints longs), this > > sucks. An open issue already exists about this: http://bugs.python.org/issue14376 -- Ned Deily, nad at acm.org From waultah at gmail.com Fri Apr 24 00:21:45 2015 From: waultah at gmail.com (Riley Banks) Date: Thu, 23 Apr 2015 23:21:45 +0100 Subject: [Python-ideas] Make range objects orderable Message-ID: I propose adding the ability to compare range objects using methods (e.g. range.issubrange) and/or regular operators. Example: In [56]: range(0, 10, 3).issubrange(range(10)) Out[56]: True In [57]: range(0, 10, 3) <= range(10) Out[57]: True In [58]: range(10) <= range(0, 10, 3) Out[58]: False I'll write a patch if you decide that this idea is worth implementing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Fri Apr 24 00:34:59 2015 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 23 Apr 2015 17:34:59 -0500 Subject: [Python-ideas] Make range objects orderable In-Reply-To: References: Message-ID: On Thu, Apr 23, 2015 at 5:21 PM, Riley Banks wrote: > I propose adding the ability to compare range objects using methods (e.g. > range.issubrange) and/or regular operators. Example: > > In [56]: range(0, 10, 3).issubrange(range(10)) > Out[56]: True > > In [57]: range(0, 10, 3) <= range(10) > Out[57]: True > > In [58]: range(10) <= range(0, 10, 3) > Out[58]: False > > I'll write a patch if you decide that this idea is worth implementing. > Could you give some examples of how or when this is necessary and why you need it? Cheers, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwblas at gmail.com Fri Apr 24 00:44:50 2015 From: dwblas at gmail.com (David Blaschke) Date: Thu, 23 Apr 2015 15:44:50 -0700 Subject: [Python-ideas] Make range objects orderable In-Reply-To: References: Message-ID: f you don't mind converting to a set (can alter the order but the original range can be easily reproduced) then print set(range(0, 10, 3)).issubset(set(range(10))) On 4/23/15, Ian Cordasco wrote: > On Thu, Apr 23, 2015 at 5:21 PM, Riley Banks wrote: > >> I propose adding the ability to compare range objects using methods (e.g. >> range.issubrange) and/or regular operators. Example: >> >> In [56]: range(0, 10, 3).issubrange(range(10)) >> Out[56]: True >> >> In [57]: range(0, 10, 3) <= range(10) >> Out[57]: True >> >> In [58]: range(10) <= range(0, 10, 3) >> Out[58]: False >> >> I'll write a patch if you decide that this idea is worth implementing. >> > > Could you give some examples of how or when this is necessary and why you > need it? > > Cheers, > Ian > -- With the simplicity of true nature, there shall be no desire. Without desire, one's original nature will be at peace. And the world will naturally be in accord with the right Way. Tao Te Ching From ethan at stoneleaf.us Fri Apr 24 02:15:21 2015 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 23 Apr 2015 17:15:21 -0700 Subject: [Python-ideas] Make range objects orderable In-Reply-To: References: Message-ID: <20150424001521.GB32422@stoneleaf.us> On 04/23, Riley Banks wrote: > I propose adding the ability to compare range objects using methods (e.g. > range.issubrange) and/or regular operators. Example: > > In [56]: range(0, 10, 3).issubrange(range(10)) > Out[56]: True > > In [57]: range(0, 10, 3) <= range(10) > Out[57]: True > > In [58]: range(10) <= range(0, 10, 3) > Out[58]: False I seem to recall orderable ranges being rejected before. For example, is less-than, or subrange, dependent on the values themselves, or on the lowest and highest values, or on the start and end value, or ... In other words, given: a = range(11) b = range(12, step=3) is a < b, or b < a? Why? -- ~Ethan~ From python at mrabarnett.plus.com Fri Apr 24 02:31:05 2015 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 24 Apr 2015 01:31:05 +0100 Subject: [Python-ideas] Make range objects orderable In-Reply-To: <20150424001521.GB32422@stoneleaf.us> References: <20150424001521.GB32422@stoneleaf.us> Message-ID: <55398EC9.4010408@mrabarnett.plus.com> On 2015-04-24 01:15, Ethan Furman wrote: > On 04/23, Riley Banks wrote: > >> I propose adding the ability to compare range objects using methods (e.g. >> range.issubrange) and/or regular operators. Example: >> >> In [56]: range(0, 10, 3).issubrange(range(10)) >> Out[56]: True >> >> In [57]: range(0, 10, 3) <= range(10) >> Out[57]: True >> >> In [58]: range(10) <= range(0, 10, 3) >> Out[58]: False > > I seem to recall orderable ranges being rejected before. For example, is > less-than, or subrange, dependent on the values themselves, or on the > lowest and highest values, or on the start and end value, or ... > > In other words, given: > > a = range(11) > b = range(12, step=3) > > is a < b, or b < a? Why? > It could be functionally equivalent to list(a) < list(b). From tjreedy at udel.edu Fri Apr 24 03:11:32 2015 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Apr 2015 21:11:32 -0400 Subject: [Python-ideas] Make range objects orderable In-Reply-To: <55398EC9.4010408@mrabarnett.plus.com> References: <20150424001521.GB32422@stoneleaf.us> <55398EC9.4010408@mrabarnett.plus.com> Message-ID: On 4/23/2015 8:31 PM, MRAB wrote: > On 2015-04-24 01:15, Ethan Furman wrote: >> On 04/23, Riley Banks wrote: >> >>> I propose adding the ability to compare range objects using methods >>> (e.g. >>> range.issubrange) and/or regular operators. Example: >>> >>> In [56]: range(0, 10, 3).issubrange(range(10)) >>> Out[56]: True >>> >>> In [57]: range(0, 10, 3) <= range(10) >>> Out[57]: True This was not obvious to me. Ranges are not sets. >>> In [58]: range(10) <= range(0, 10, 3) >>> Out[58]: False >> >> I seem to recall orderable ranges being rejected before. For example, is >> less-than, or subrange, dependent on the values themselves, or on the >> lowest and highest values, or on the start and end value, or ... I remember the same. There is no intuitive answer. Let us not rehash this again. >> In other words, given: >> >> a = range(11) >> b = range(12, step=3) >> >> is a < b, or b < a? Why? >> > It could be functionally equivalent to list(a) < list(b). For range(0, 10, 3) <= range(10), this would give the opposite answer as the set interpretation. -- Terry Jan Reedy From rob.cliffe at btinternet.com Fri Apr 24 13:25:25 2015 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 24 Apr 2015 12:25:25 +0100 Subject: [Python-ideas] Issue with sys.exit and a long In-Reply-To: References: Message-ID: <553A2825.8080905@btinternet.com> On 23/04/2015 19:52, Ryan Gonzalez wrote: > Observe: > > ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1)' > ryan at DevPC-LX:~/bfort$ python -c 'import sys; sys.exit(1L)' > 1 > ryan at DevPC-LX:~/bfort$ > > If I call sys.exit with a long value under Python 2, it prints it. I > count this as a bug because: > > 1. int and long are supposed to be as similar as possible. > 2. When you're using Hy (which implicitly makes all ints longs), this > sucks. > Yes the same happens from the [2.7.3] interpreter if you exit from it with import sys; sys.exit(0L) > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > No virus found in this message. > Checked by AVG - www.avg.com > Version: 2014.0.4800 / Virus Database: 4311/9607 - Release Date: 04/23/15 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Fri Apr 24 16:50:14 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 24 Apr 2015 11:50:14 -0300 Subject: [Python-ideas] Make range objects orderable In-Reply-To: References: <20150424001521.GB32422@stoneleaf.us> <55398EC9.4010408@mrabarnett.plus.com> Message-ID: Sorry - it looks like Guido's time machine have been there. Since it does not mean anything (as in meaningless) to compare ranges without specifying a "key", and since ordering functions in Python do allow a "key" - it looks like this problem is already resolved, whatever the need: >>> sorted([range(10), range(1,14, 3)], key=len) [range(1, 14, 3), range(0, 10)] >>> sorted([range(10), range(1,14, 3)], key=max) [range(0, 10), range(1, 14, 3)] >>> sorted([range(10), range(1,14, 3)], key=min) [range(0, 10), range(1, 14, 3)] >>> On 23 April 2015 at 22:11, Terry Reedy wrote: > On 4/23/2015 8:31 PM, MRAB wrote: >> >> On 2015-04-24 01:15, Ethan Furman wrote: >>> >>> On 04/23, Riley Banks wrote: >>> >>>> I propose adding the ability to compare range objects using methods >>>> (e.g. >>>> range.issubrange) and/or regular operators. Example: >>>> >>>> In [56]: range(0, 10, 3).issubrange(range(10)) >>>> Out[56]: True >>>> >>>> In [57]: range(0, 10, 3) <= range(10) >>>> Out[57]: True > > > This was not obvious to me. Ranges are not sets. > >>>> In [58]: range(10) <= range(0, 10, 3) >>>> Out[58]: False >>> >>> >>> I seem to recall orderable ranges being rejected before. For example, is >>> less-than, or subrange, dependent on the values themselves, or on the >>> lowest and highest values, or on the start and end value, or ... > > > I remember the same. There is no intuitive answer. Let us not rehash this > again. > >>> In other words, given: >>> >>> a = range(11) >>> b = range(12, step=3) >>> >>> is a < b, or b < a? Why? >>> >> It could be functionally equivalent to list(a) < list(b). > > > For range(0, 10, 3) <= range(10), this would give the opposite answer as the > set interpretation. > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From njs at pobox.com Sat Apr 25 03:03:35 2015 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 24 Apr 2015 18:03:35 -0700 Subject: [Python-ideas] concurrency-local namespaces In-Reply-To: <5537EDAF.2030300@gmail.com> References: <5537EA5A.9000701@gmail.com> <5537EDAF.2030300@gmail.com> Message-ID: On Wed, Apr 22, 2015 at 11:51 AM, Yury Selivanov wrote: > Guido, Nathaniel, > > On 2015-04-22 2:45 PM, Guido van Rossum wrote: >> >> I'll have to leave this up to you and Nathaniel and others to ponder, I've >> run out of time to think about more issues. Sorry! (I do think that the >> conclusion of that patch was to go with current_task() intead.) > > > Yes, that was the conclusion. Because I was interested to add > this functionality for web frameworks, where, as Nathaniel > mentioned, you control the top-level Task object. So I gave up > on pushing the 'context' idea. > > If we want to add a generic thread-local-like object to > coroutines, I think that it has to be done on an asyncio > event loop level. Because event loop is not just about > coroutines, it's also about callbacks, and it is useful > to *avoid* loosing context in them. > > Nathaniel, I'd be glad if you could glance over my old proposal. > It's based on ideas I expressed in this thread: > https://groups.google.com/forum/#!topic/python-tulip/zix5HQxtElg Hi Yury, Unfortunately I haven't had cause to write any async code in the last ~5 years or so, so my knowledge is all out of date and I'm not up on the terminology there. With that in mind... I have two fairly basic questions after reading your proposal: 1) Can everyone really agree on how you fetch the current "task context" in an async framework? E.g., your proposal talks about looking at attributes on the tulip main loop object. For a project like numpy that doesn't involve any async stuff itself, but is regularly used alongside multiple existing frameworks (tornado, twisted, asyncio, etc.), is there some standard way to find "the main loop" that works on all of them? 2) How fast can we make accessing "task-local storage"? The reason I ask is, again, numpy, where we essentially have a single word-sized bitmask (corresponding to the IEEE 754 control word) that we need to stash somewhere and then access on every single arithmetic operation. This means that the cost of accessing this thing is a non-trivial speed bottleneck. Classic thread-local storage is great for this, because it's heavily optimized by all common platforms, to the point where accessing it is basically just following a pointer or two. Any API that involves calling Python methods on every access is definitely not going to fly here -- especially since this would be additional overhead on *every* user of numpy, even the vast majority of users who aren't writing any async code at all. These questions make me wonder if it wouldn't be better to focus on fixing 'with' instead, since it moves the problem closer to the actual affected code -- in particular, a context manager doesn't need to know anything about mainloops to manage state, and it doesn't have any impact on people who aren't using A straw man proposal would be something like: ----- Currently, the code with EXPR as VAR: PARTIAL-BLOCK-1 f((yield foo)) PARTIAL-BLOCK-2 is equivalent to (cribbing from PEP 343): mgr = (EXPR) exit = type(mgr).__exit__ # Not calling it yet value = type(mgr).__enter__(mgr) exc = True try: try: VAR = value # Only if "as VAR" is present PARTIAL-BLOCK-1 f((yield foo)) PARTIAL-BLOCK-2 except: exc = False if not exit(mgr, *sys.exc_info()): raise finally: if exc: exit(mgr, None, None, None) Starting in 3.xx, two new methods __suspend__ and __resume__ are added to the context manager protocol, and the above code instead becomes: mgr = (EXPR) exit = type(mgr).__exit__ # Not calling it yet #### suspend = getattr(type(mgr), "__suspend__", lambda: None) resume = getattr(type(mgr), "__resume__", lambda: None) #### value = type(mgr).__enter__(mgr) exc = True try: try: VAR = value # Only if "as VAR" is present PARTIAL-BLOCK-1 #### suspend() tmp = yield foo resume() f(tmp) #### PARTIAL-BLOCK-2 except: exc = False if not exit(mgr, *sys.exc_info()): raise finally: if exc: exit(mgr, None, None, None) If a yield occurs inside multiple nested 'with' blocks, then they are __suspend__'ed from the inside out, and __resume__'ed from the outside in. 'yield from' can be reduced to 'yield' (as described in PEP 380), and then the above transformation is applied. Note that this is orthogonal to PEP 492, because the problem being solved is caused by any 'with' block containing a yield statement. The proposed 'async with' syntax handles a different issue, where __enter__ and __exit__ may need to yield. If PEP 492 is accepted, then analogous changes should also be made to the 'async with' syntax (e.g., by adding __asuspend__ and __aresume__ to the async context manager protocol). ----- If there's interest I can throw the above into a quick PEP. Unfortunately I don't have the bandwidth to implement it (recall that I don't actually use asyncio :-)), nor the relevant knowledge of the compiler -- though I *think* there shouldn't be any particularly nasty problems in implementing it, since the compiler has static knowledge of the presence of all yield points? In particular this means that even the getattr() calls at the top can be optimized out for any 'with' blocks that don't actually contain any yield points, so the proposal has zero impact on non-async code. For async code, the main downside is that this could make yield's even more expensive than they currently are -- but this is only when within a 'with' block, and I suspect that this cost is small in any case. -n -- Nathaniel J. Smith -- http://vorpus.org From ncoghlan at gmail.com Sat Apr 25 06:02:24 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Apr 2015 14:02:24 +1000 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: References: Message-ID: On 18 April 2015 at 16:05, Andrew Barnert wrote: > On Apr 17, 2015, at 21:48, Rustom Mody wrote: >> >> On Sat, Apr 18, 2015 at 12:28 AM, Yury Selivanov >> wrote: >>> Hello python-ideas, >>> >>> Here's my proposal to add async/await in Python. >>> >>> I believe that PEPs 380 and 3156 were a major breakthrough for Python 3, >> >> I am also interested in this topic --- from the other side. >> As a teacher of python it is my finding that the >> terminology/documentation around generators is rather chaotic and >> messy. >> Basically given: >> def foo(): >> yield 1 >> bar = foo() >> >> what do we call foo and what do we call bar? >> It is suggested that foo is "generator-function" and bar is "generator-object" >> Unfortunately python does not aid this distinction; witness >> >>>>> def foo(): >> ... yield 1 >> ... >>>>> bar = foo() >>>>> type(foo) >> >>>>> type(bar) >> > > I assume you're ok with the type(bar); the type of a generator object is named "generator", just like the type of a bound method object is named "method" and the type of an integer object is named "int", and so on. > > So presumably you don't like the former. But what about it? > > The only thing I can imagine is the fact that, even though a generator function is conceptually a "subkind" of function, there is no generator function subtype of the function type? > > But so what? There's no closure function subtype, just functions whose closure is nonempty; there's no positive int subtype, just ints whose value is positive; etc. And likewise, there's no callable supertype that function and method (and other things) inherit. > > If there were some additional API that a generator function should provide that other functions don't (or some implementation reason that makes it worth subclassing for convenience, I suppose), that would be a good reason for needing a subtype. For example, generator a subtype of iterator, just as file is, because they both add new methods that don't make sense for the base iterator type. > > But just wanting repr(type(x)) to give you more information about x, that's not a good reason to add a subtype. > > So, it seems to me like both of these are returning something you should reasonably expect. Catching up on email following PyCon travel, there's a reasonable question here related to the repr() of "foo" itself: >>> def foo(): ... yield 1 ... >>> foo >>> import dis >>> dis.show_code(foo) Name: foo Filename: Argument count: 0 Kw-only arguments: 0 Number of locals: 0 Stack size: 1 Flags: OPTIMIZED, NEWLOCALS, GENERATOR, NOFREE Constants: 0: None 1: 1 Modifying the repr of a function based on whether or not the GENERATOR flag was set on the code object would be quite doable. For example, the above could be: >>> foo type(foo) would still be unchanged, we'd just be exposing some additional info in the value repr(). While it's a less significant behavioural difference than being a generator function, a non-empty closure could conceivably also be reported in the repr(): Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From robertc at robertcollins.net Sat Apr 25 06:15:50 2015 From: robertc at robertcollins.net (Robert Collins) Date: Sat, 25 Apr 2015 16:15:50 +1200 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: References: Message-ID: On 25 April 2015 at 16:02, Nick Coghlan wrote: > While it's a less significant behavioural difference than being a > generator function, a non-empty closure could conceivably also be > reported in the repr(): > > > I like this suggestion. Doing it for the await changes as well would be great. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From ncoghlan at gmail.com Sat Apr 25 06:18:47 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Apr 2015 14:18:47 +1000 Subject: [Python-ideas] Make range objects orderable In-Reply-To: References: <20150424001521.GB32422@stoneleaf.us> <55398EC9.4010408@mrabarnett.plus.com> Message-ID: On 25 April 2015 at 00:50, Joao S. O. Bueno wrote: > Since it does not mean anything (as in meaningless) to compare ranges without > specifying a "key", and since ordering functions in Python do allow a > "key" - it looks > like this problem is already resolved, whatever the need: > >>>> sorted([range(10), range(1,14, 3)], key=len) > [range(1, 14, 3), range(0, 10)] >>>> sorted([range(10), range(1,14, 3)], key=max) > [range(0, 10), range(1, 14, 3)] >>>> sorted([range(10), range(1,14, 3)], key=min) > [range(0, 10), range(1, 14, 3)] ranges() are actually defined as memory-efficient tuples these days (https://docs.python.org/3/library/stdtypes.html#ranges), and have supported "==" and "!=" using sequence comparison semantics since 3.3, so there's a defined natural order for them these days (the ordering of the equivalent tuples). That said, I'm still not sure it makes sense to allow ordering them. The cases I can think of where it might make sense (such as ordering graph nodes) would all use an actual tuple instead. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Apr 25 06:51:43 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Apr 2015 21:51:43 -0700 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: References: Message-ID: +1 On Fri, Apr 24, 2015 at 9:15 PM, Robert Collins wrote: > On 25 April 2015 at 16:02, Nick Coghlan wrote: > > > While it's a less significant behavioural difference than being a > > generator function, a non-empty closure could conceivably also be > > reported in the repr(): > > > > > > > > I like this suggestion. Doing it for the await changes as well would be > great. > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Apr 25 06:52:48 2015 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Apr 2015 21:52:48 -0700 Subject: [Python-ideas] Make range objects orderable In-Reply-To: References: <20150424001521.GB32422@stoneleaf.us> <55398EC9.4010408@mrabarnett.plus.com> Message-ID: On Fri, Apr 24, 2015 at 9:18 PM, Nick Coghlan wrote: > On 25 April 2015 at 00:50, Joao S. O. Bueno wrote: > > Since it does not mean anything (as in meaningless) to compare ranges > without > > specifying a "key", and since ordering functions in Python do allow a > > "key" - it looks > > like this problem is already resolved, whatever the need: > > > >>>> sorted([range(10), range(1,14, 3)], key=len) > > [range(1, 14, 3), range(0, 10)] > >>>> sorted([range(10), range(1,14, 3)], key=max) > > [range(0, 10), range(1, 14, 3)] > >>>> sorted([range(10), range(1,14, 3)], key=min) > > [range(0, 10), range(1, 14, 3)] > > ranges() are actually defined as memory-efficient tuples these days > (https://docs.python.org/3/library/stdtypes.html#ranges), and have > supported "==" and "!=" using sequence comparison semantics since 3.3, > so there's a defined natural order for them these days (the ordering > of the equivalent tuples). > > That said, I'm still not sure it makes sense to allow ordering them. > The cases I can think of where it might make sense (such as ordering > graph nodes) would all use an actual tuple instead. > Seems everybody has their own view on how range() objects should be ordered. That's one good reason against a builtin ordering. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Apr 25 08:16:43 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Apr 2015 16:16:43 +1000 Subject: [Python-ideas] Generator/Coroutiine Ontology (was async/await in Python) In-Reply-To: References: Message-ID: On 25 April 2015 at 14:51, Guido van Rossum wrote: > +1 RFE filed as http://bugs.python.org/issue24056 Cheers, Nick. > > On Fri, Apr 24, 2015 at 9:15 PM, Robert Collins > wrote: >> >> On 25 April 2015 at 16:02, Nick Coghlan wrote: >> >> > While it's a less significant behavioural difference than being a >> > generator function, a non-empty closure could conceivably also be >> > reported in the repr(): >> > >> > >> > >> >> I like this suggestion. Doing it for the await changes as well would be >> great. >> >> -Rob >> >> -- >> Robert Collins >> Distinguished Technologist >> HP Converged Cloud >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sat Apr 25 18:14:57 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 25 Apr 2015 19:14:57 +0300 Subject: [Python-ideas] Bytestrings in Python 2 Message-ID: Here is an idea that perhaps will help to prepare Python 2 code for converting to Python 3. Currently bytes is just an alias of str in Python 2, and the "b" prefix of string literals is ignored. There are no differences between natural strings and bytes. I propose to add special bit to str instances and set it for bytes literals and strings created from binary sources (read from binary files, received from sockets, the result of unicode.encode() and struct.pack(), etc). With -3 flag operations with binary strings that doesn't allowed for bytes in Python 3 (e.g. encoding or coercing to unicode) will emit a warning. Unfortunately we can't change the bytes constructor in minor version, it should left an alias to str in 2.7. So the result of bytes() will be not tagged as binary string. May be it is too late for this. From markus at unterwaditzer.net Sat Apr 25 21:27:36 2015 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Sat, 25 Apr 2015 21:27:36 +0200 Subject: [Python-ideas] Bytestrings in Python 2 In-Reply-To: References: Message-ID: <20150425192736.GA15222@untibox.unti> On Sat, Apr 25, 2015 at 07:14:57PM +0300, Serhiy Storchaka wrote: > Here is an idea that perhaps will help to prepare Python 2 code for > converting to Python 3. > > Currently bytes is just an alias of str in Python 2, and the "b" prefix of > string literals is ignored. There are no differences between natural strings > and bytes. I propose to add special bit to str instances and set it for > bytes literals and strings created from binary sources (read from binary > files, received from sockets, the result of unicode.encode() and > struct.pack(), etc). With -3 flag operations with binary strings that > doesn't allowed for bytes in Python 3 (e.g. encoding or coercing to unicode) > will emit a warning. Unfortunately we can't change the bytes constructor in > minor version, it should left an alias to str in 2.7. So the result of > bytes() will be not tagged as binary string. > > May be it is too late for this. You can get similar kinds of warnings with unicode-nazi (https://github.com/mitsuhiko/unicode-nazi), so I'm not sure if this would be that helpful. -- Markus From ncoghlan at gmail.com Sun Apr 26 05:02:32 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Apr 2015 13:02:32 +1000 Subject: [Python-ideas] Bytestrings in Python 2 In-Reply-To: <20150425192736.GA15222@untibox.unti> References: <20150425192736.GA15222@untibox.unti> Message-ID: On 26 April 2015 at 05:27, Markus Unterwaditzer wrote: > On Sat, Apr 25, 2015 at 07:14:57PM +0300, Serhiy Storchaka wrote: >> Here is an idea that perhaps will help to prepare Python 2 code for >> converting to Python 3. >> >> Currently bytes is just an alias of str in Python 2, and the "b" prefix of >> string literals is ignored. There are no differences between natural strings >> and bytes. I propose to add special bit to str instances and set it for >> bytes literals and strings created from binary sources (read from binary >> files, received from sockets, the result of unicode.encode() and >> struct.pack(), etc). With -3 flag operations with binary strings that >> doesn't allowed for bytes in Python 3 (e.g. encoding or coercing to unicode) >> will emit a warning. Unfortunately we can't change the bytes constructor in >> minor version, it should left an alias to str in 2.7. So the result of >> bytes() will be not tagged as binary string. >> >> May be it is too late for this. > > You can get similar kinds of warnings with unicode-nazi > (https://github.com/mitsuhiko/unicode-nazi), so I'm not sure if this would be > that helpful. Mentioning that utility in the porting guide could potentially be useful, but I don't think it's a substitute for Serhiy's suggestion here. Serhiy's suggestion covers a slightly different situation, which is that we can't warn about the following code snippet in Python 2, even though we know bytes objects don't have an encode method in Python 3: "str".encode(encoding) The reason is that we can't easily tell the difference between something that is correct in both Python 2 & 3 like: "text".encode("utf-8") (str->str encoding in Python 2, str->bytes encoding in Python 3) and something that will break in Python 3 like: "data".encode("hex") The single source version of the latter is actually 'codecs.encode(b"data", "hex")', but it's quite hard for an analyser or test suite to pick that up and recommend the change, as it's hard to tell the difference between "str-as-text-object" and "str-as-binary-data-object" in Python 2. Looking at the way string objects are stored in Python 2, it's possible that the ob_sstate field (which tracks the interning status of string instances) could potentially be co-opted to hold this additional flag. If the new flag was *only* set when "-3" was specified, then there'd only be a potential compatibility risk in that context (the PyString_CHECK_INTERNED macro currently assumes that a non-zero value in ob_sstate always indicates an interned string). There'd be a more general performance risk however, as PyString_CHECK_INTERNED would also need to be updated to either mask out the new "this is probably binary data" state flag unconditionally, or else to check the Py3k warning flag and mask out the new flag conditionally. Either way, we'd be making "is this interned or not?" checks slightly more expensive and the interpreter does a *lot* of those. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sun Apr 26 07:14:56 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 26 Apr 2015 08:14:56 +0300 Subject: [Python-ideas] Bytestrings in Python 2 In-Reply-To: <20150425192736.GA15222@untibox.unti> References: <20150425192736.GA15222@untibox.unti> Message-ID: On 25.04.15 22:27, Markus Unterwaditzer wrote: > On Sat, Apr 25, 2015 at 07:14:57PM +0300, Serhiy Storchaka wrote: >> Here is an idea that perhaps will help to prepare Python 2 code for >> converting to Python 3. >> >> Currently bytes is just an alias of str in Python 2, and the "b" prefix of >> string literals is ignored. There are no differences between natural strings >> and bytes. I propose to add special bit to str instances and set it for >> bytes literals and strings created from binary sources (read from binary >> files, received from sockets, the result of unicode.encode() and >> struct.pack(), etc). With -3 flag operations with binary strings that >> doesn't allowed for bytes in Python 3 (e.g. encoding or coercing to unicode) >> will emit a warning. Unfortunately we can't change the bytes constructor in >> minor version, it should left an alias to str in 2.7. So the result of >> bytes() will be not tagged as binary string. >> >> May be it is too late for this. > > You can get similar kinds of warnings with unicode-nazi > (https://github.com/mitsuhiko/unicode-nazi), so I'm not sure if this would be > that helpful. Not quite. With unicode-nazi 'foo' == u'foo' emits a warning, with my proposition it doesn't, but b'foo' == u'foo' does. unicode-nazi should produce a lot of warnings in the stdlib and user code, my proposition should be more compatible. From storchaka at gmail.com Sun Apr 26 07:38:23 2015 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 26 Apr 2015 08:38:23 +0300 Subject: [Python-ideas] Bytestrings in Python 2 In-Reply-To: References: <20150425192736.GA15222@untibox.unti> Message-ID: On 26.04.15 06:02, Nick Coghlan wrote: > Serhiy's suggestion covers a slightly different situation, which is > that we can't warn about the following code snippet in Python 2, even > though we know bytes objects don't have an encode method in Python 3: > > "str".encode(encoding) > > The reason is that we can't easily tell the difference between > something that is correct in both Python 2 & 3 like: > > "text".encode("utf-8") > > (str->str encoding in Python 2, str->bytes encoding in Python 3) > > and something that will break in Python 3 like: > > "data".encode("hex") > > The single source version of the latter is actually > 'codecs.encode(b"data", "hex")', but it's quite hard for an analyser > or test suite to pick that up and recommend the change, as it's hard > to tell the difference between "str-as-text-object" and > "str-as-binary-data-object" in Python 2. A warning about "data".encode("hex") can be implemented without this change. "hex" is not text encoding, and we can add special flags for text and binary encodings, and emits a warning if binary encoding is used in str.encode() (but not in codecs.encode()). My suggestion covers a case of b"str".encode(encoding) > Looking at the way string objects are stored in Python 2, it's > possible that the ob_sstate field (which tracks the interning status > of string instances) could potentially be co-opted to hold this > additional flag. If the new flag was *only* set when "-3" was > specified, then there'd only be a potential compatibility risk in that > context (the PyString_CHECK_INTERNED macro currently assumes that a > non-zero value in ob_sstate always indicates an interned string). > > There'd be a more general performance risk however, as > PyString_CHECK_INTERNED would also need to be updated to either mask > out the new "this is probably binary data" state flag unconditionally, > or else to check the Py3k warning flag and mask out the new flag > conditionally. Either way, we'd be making "is this interned or not?" > checks slightly more expensive and the interpreter does a *lot* of > those. PyString_CHECK_INTERNED is used only in 8 places in CPython and unconditional masking out shouldn't be too expensive in comparing with surrounding code. But PyString_CHECK_INTERNED can be used in third-party extensions, and this change breaks binary compatibility. So we can't change PyString_CHECK_INTERNED in bugfix release. But we can use a byte after the terminated null byte in a string. From abarnert at yahoo.com Sun Apr 26 11:22:05 2015 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 26 Apr 2015 02:22:05 -0700 Subject: [Python-ideas] Bytestrings in Python 2 In-Reply-To: References: <20150425192736.GA15222@untibox.unti> Message-ID: On Apr 25, 2015, at 20:02, Nick Coghlan wrote: > > On 26 April 2015 at 05:27, Markus Unterwaditzer > wrote: >> On Sat, Apr 25, 2015 at 07:14:57PM +0300, Serhiy Storchaka wrote: >>> Here is an idea that perhaps will help to prepare Python 2 code for >>> converting to Python 3. >>> >>> Currently bytes is just an alias of str in Python 2, and the "b" prefix of >>> string literals is ignored. There are no differences between natural strings >>> and bytes. I propose to add special bit to str instances and set it for >>> bytes literals and strings created from binary sources (read from binary >>> files, received from sockets, the result of unicode.encode() and >>> struct.pack(), etc). With -3 flag operations with binary strings that >>> doesn't allowed for bytes in Python 3 (e.g. encoding or coercing to unicode) >>> will emit a warning. Unfortunately we can't change the bytes constructor in >>> minor version, it should left an alias to str in 2.7. So the result of >>> bytes() will be not tagged as binary string. >>> >>> May be it is too late for this. >> >> You can get similar kinds of warnings with unicode-nazi >> (https://github.com/mitsuhiko/unicode-nazi), so I'm not sure if this would be >> that helpful. > > Mentioning that utility in the porting guide could potentially be > useful, but I don't think it's a substitute for Serhiy's suggestion > here. > > Serhiy's suggestion covers a slightly different situation, which is > that we can't warn about the following code snippet in Python 2, even > though we know bytes objects don't have an encode method in Python 3: > > "str".encode(encoding) > > The reason is that we can't easily tell the difference between > something that is correct in both Python 2 & 3 like: > > "text".encode("utf-8") > > (str->str encoding in Python 2, str->bytes encoding in Python 3) > > and something that will break in Python 3 like: > > "data".encode("hex") I don't think that's a problem. The former is legal in both 2.x and 3.x, but it has a different meaning--in 2.x it means "decode with the system encoding, then recode to UTF-8". Unless it's called on a pure-printable-ASCII literal, there's no reason to expect that code to work without changes in 3.x. (And that's even ignoring the fact that the vast majority of calls to str.encode are bugs, either called on a variable you think is a unicode but is actually a str, or just introduced by a novice throwing in random calls to encode, decode, and str until some exception goes away.) Whether the encoding is a literal for a unicode->bytes encoding, a literal for a non-3.x-compatible encoding, or a variable whose value can't be guessed statically doesn't really matter; in every case, it's something you need to look at for porting to 3.x. So a warning still seems both doable and worth doing, whether you're talking about a static linter tool or a -3 mode that tracks probably-bytes. > The single source version of the latter is actually > 'codecs.encode(b"data", "hex")', but it's quite hard for an analyser > or test suite to pick that up and recommend the change, as it's hard > to tell the difference between "str-as-text-object" and > "str-as-binary-data-object" in Python 2. > > Looking at the way string objects are stored in Python 2, it's > possible that the ob_sstate field (which tracks the interning status > of string instances) could potentially be co-opted to hold this > additional flag. If the new flag was *only* set when "-3" was > specified, then there'd only be a potential compatibility risk in that > context (the PyString_CHECK_INTERNED macro currently assumes that a > non-zero value in ob_sstate always indicates an interned string). > > There'd be a more general performance risk however, as > PyString_CHECK_INTERNED would also need to be updated to either mask > out the new "this is probably binary data" state flag unconditionally, > or else to check the Py3k warning flag and mask out the new flag > conditionally. Either way, we'd be making "is this interned or not?" > checks slightly more expensive and the interpreter does a *lot* of > those. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From njs at pobox.com Wed Apr 29 22:22:49 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 29 Apr 2015 13:22:49 -0700 Subject: [Python-ideas] Letting context managers react to yields inside their scope Message-ID: Hi all, Well, after a few days no-one has responded to my post on another thread about this [1], but the more I thought about it the more this seemed like a good idea, so I wrote up a little more-formal proposal (attached) for letting context managers react to 'yield's that occur within their 'with' block. This should in many ways be treated as a straw man proposal -- there's tons I don't know about how async code is written in Python these days -- but it seems like a good idea to me and I'd like to hear what everyone else thinks :-). -n [1] https://mail.python.org/pipermail/python-ideas/2015-April/033176.html -- Nathaniel J. Smith -- http://vorpus.org -------------- next part -------------- A non-text attachment was scrubbed... Name: with-yield-draft-pep.rst Type: text/x-rst Size: 11843 bytes Desc: not available URL: From guido at python.org Wed Apr 29 22:36:14 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Apr 2015 13:36:14 -0700 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: Message-ID: This seems reasonable, though mostly also non-urgent. Have you thought about how it interacts with PEP 492 yet? On Wed, Apr 29, 2015 at 1:22 PM, Nathaniel Smith wrote: > Hi all, > > Well, after a few days no-one has responded to my post on another > thread about this [1], but the more I thought about it the more this > seemed like a good idea, so I wrote up a little more-formal proposal > (attached) for letting context managers react to 'yield's that occur > within their 'with' block. > > This should in many ways be treated as a straw man proposal -- there's > tons I don't know about how async code is written in Python these days > -- but it seems like a good idea to me and I'd like to hear what > everyone else thinks :-). > > -n > > [1] https://mail.python.org/pipermail/python-ideas/2015-April/033176.html > > -- > Nathaniel J. Smith -- http://vorpus.org > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Apr 29 23:01:56 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 29 Apr 2015 17:01:56 -0400 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: Message-ID: <554146C4.7010704@gmail.com> Hi Nathanial, Sorry for not replying to your last email promptly. FWIW I was going to do it later today ;) My opinion on this subject (and I've implemented lots of local-context kind of objects in different frameworks) is that *just* inserting some kind of suspend/resume points before/after yields does not work. Why: 1. Greenlets, gevent, eventlet, stackless python: they do not have 'yield'. Context switches are invisible to the interpreter. And those frameworks are popular. You want numpy.errstate/decimal.localcontext to work there too. 2. I'm curious how this will be implemented and what's the performance impact. 3. I think that mechanism should be more generic than just 'with' staments. What if you want to have a decorator that applies some context? What if you can't write it as a generator with @contextlib.contextmanager? What I would propose: I think that the right approach here would be to have a standard protocol; something similar to how we defined WSGI -- it doesn't require any changes in the interpreter, it is a protocol. For instance, we could have a module in the stdlib, with a class Context: class Context: @classmethod def __suspend_contexts(cls): for child in cls.__subclasses__(): child.__suspend_context() @classmethod def __resume_contexts(cls): .. To inform context objects that the context is about to change, asyncio event loop would call Context.__suspend_contexts() I think that it's up to the event loop/library/framework to manage context switches properly. In asyncio, for instance, you can attach callbacks to Futures. I think it would be great if such callbacks are executed in the right context. I would also suggest to think about a universal context -- the one that works both for asyncio coroutines and threadpools they might use. This way any framework can implement the context in the most efficient way. All in all, I'm in favor of API and a couple functions added to the stdlib for this. Thanks, Yury On 2015-04-29 4:22 PM, Nathaniel Smith wrote: > Hi all, > > Well, after a few days no-one has responded to my post on another > thread about this [1], but the more I thought about it the more this > seemed like a good idea, so I wrote up a little more-formal proposal > (attached) for letting context managers react to 'yield's that occur > within their 'with' block. > > This should in many ways be treated as a straw man proposal -- there's > tons I don't know about how async code is written in Python these days > -- but it seems like a good idea to me and I'd like to hear what > everyone else thinks :-). > > -n > > [1] https://mail.python.org/pipermail/python-ideas/2015-April/033176.html > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From yselivanov.ml at gmail.com Wed Apr 29 23:29:10 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 29 Apr 2015 17:29:10 -0400 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: Message-ID: <55414D26.7020602@gmail.com> Nathanial, On 2015-04-29 4:22 PM, Nathaniel Smith wrote: > Interaction with PEP 492 > ======================== > > PEP 492 currently proposes an ``async with`` statement. Despite the > name, this is actually orthogonal to the proposal here: this PEP > proposes to allow context managers to react to suspensions inside > their scope, while ``async with`` makes context manager ``__enter__`` > and ``__exit__`` methods into potential ``yield`` points. If both > PEPs are accepted then we will presumably also want to add some kind > of ``__asuspend__`` and ``__aresume__`` methods to PEP 492-style > asynchronous context managers. I doubt that we'll need __asuspend__. There shouldn't be a need to perform any blocking operations there. __suspend__ and __resume__ would be enough. Yury From ncoghlan at gmail.com Thu Apr 30 02:53:19 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Apr 2015 10:53:19 +1000 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: <554146C4.7010704@gmail.com> References: <554146C4.7010704@gmail.com> Message-ID: On 30 April 2015 at 07:01, Yury Selivanov wrote: > I think that the right approach here would be to have a > standard protocol; something similar to how we defined > WSGI -- it doesn't require any changes in the interpreter, > it is a protocol. +1 - I think we have reasonably good precedent for this in the form of atexit and there's also a proposal for an "atfork" manager: https://bugs.python.org/issue16500 Decimal contexts and (to a much lesser degree) fpectl are also interesting use cases, as is the event loop policy management in asyncio. One of the more interesting aspects of a context is its scope: * Can the same context be safely accessed from multiple threads concurrently? At different points in time? * Is there a "default context" which applies if no other context has been explicitly activated? By defining a global metacontext, it becomes feasible to sensibly manage things like the active asyncio event loop policy, the decimal context, etc, in ways that multiple frameworks can interoperate with. It's also potentially something that the logging module, pdb, and other tools could integrate with, by being able to better report the active context when requested. https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack is also an interest data point, as I can easily envision a variant of that API model that remembers what contexts *were* on the stack (this becoming a "ContextStack", rather than an "ExitStack"), making it easier to switch *between* stacks, rather than throwing them away when you're done. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Thu Apr 30 05:28:45 2015 From: guido at python.org (Guido van Rossum) Date: Wed, 29 Apr 2015 20:28:45 -0700 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: <554146C4.7010704@gmail.com> References: <554146C4.7010704@gmail.com> Message-ID: On Wed, Apr 29, 2015 at 2:01 PM, Yury Selivanov wrote: > Sorry for not replying to your last email promptly. > FWIW I was going to do it later today ;) > > My opinion on this subject (and I've implemented lots > of local-context kind of objects in different frameworks) is > that *just* inserting some kind of suspend/resume points > before/after yields does not work. > Very good observation. (I.e. I hadn't thought of this myself. :-) > Why: > > 1. Greenlets, gevent, eventlet, stackless python: they do > not have 'yield'. Context switches are invisible to the > interpreter. And those frameworks are popular. You want > numpy.errstate/decimal.localcontext to work there too. > > 2. I'm curious how this will be implemented and what's > the performance impact. > > 3. I think that mechanism should be more generic than > just 'with' staments. What if you want to have a decorator > that applies some context? What if you can't write it as > a generator with @contextlib.contextmanager? > > What I would propose: > > I think that the right approach here would be to have a > standard protocol; something similar to how we defined > WSGI -- it doesn't require any changes in the interpreter, > it is a protocol. > > For instance, we could have a module in the stdlib, with a > class Context: > > class Context: > @classmethod > def __suspend_contexts(cls): > for child in cls.__subclasses__(): > child.__suspend_context() > > @classmethod > def __resume_contexts(cls): .. > > > To inform context objects that the context is about to change, > asyncio event loop would call Context.__suspend_contexts() > > I think that it's up to the event loop/library/framework > to manage context switches properly. In asyncio, for instance, > you can attach callbacks to Futures. I think it would be > great if such callbacks are executed in the right context. > I'm not sure about this, since currently those callbacks are how task scheduling happens. :-) > I would also suggest to think about a universal context -- > the one that works both for asyncio coroutines and > threadpools they might use. > > This way any framework can implement the context in the > most efficient way. > > All in all, I'm in favor of API and a couple functions > added to the stdlib for this. > I'm curious -- how would you access the current contexts? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu Apr 30 05:29:45 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 29 Apr 2015 21:29:45 -0600 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: <554146C4.7010704@gmail.com> Message-ID: On Wed, Apr 29, 2015 at 6:53 PM, Nick Coghlan wrote: > +1 - I think we have reasonably good precedent for this in the form of > atexit and there's also a proposal for an "atfork" manager: > https://bugs.python.org/issue16500 > > Decimal contexts and (to a much lesser degree) fpectl are also > interesting use cases, as is the event loop policy management in > asyncio. > > One of the more interesting aspects of a context is its scope: > > * Can the same context be safely accessed from multiple threads > concurrently? At different points in time? > * Is there a "default context" which applies if no other context has > been explicitly activated? I've thought about these and other issues for quite some time. :) I have an old implementation of a generic context type kicking around that came out of my initial work on the "Import Engine" PEP. At the time it seemed genuinely useful and would have been used in decimal and maybe for email policies (in addition to import system contexts). I'm spinning back up on the successor to that PEP, which is part of why I brought "async-local" contexts the other day. > > By defining a global metacontext, it becomes feasible to sensibly > manage things like the active asyncio event loop policy, the decimal > context, etc, in ways that multiple frameworks can interoperate with. > It's also potentially something that the logging module, pdb, and > other tools could integrate with, by being able to better report the > active context when requested. +1 > > https://docs.python.org/3/library/contextlib.html#contextlib.ExitStack > is also an interest data point, as I can easily envision a variant of > that API model that remembers what contexts *were* on the stack (this > becoming a "ContextStack", rather than an "ExitStack"), making it > easier to switch *between* stacks, rather than throwing them away when > you're done. interesting. -eric From ericsnowcurrently at gmail.com Thu Apr 30 06:04:03 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 29 Apr 2015 22:04:03 -0600 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: <554146C4.7010704@gmail.com> References: <554146C4.7010704@gmail.com> Message-ID: On Wed, Apr 29, 2015 at 3:01 PM, Yury Selivanov wrote: > Hi Nathanial, > > Sorry for not replying to your last email promptly. > FWIW I was going to do it later today ;) > > My opinion on this subject (and I've implemented lots > of local-context kind of objects in different frameworks) is > that *just* inserting some kind of suspend/resume points > before/after yields does not work. Good point. > > Why: > > 1. Greenlets, gevent, eventlet, stackless python: they do > not have 'yield'. Context switches are invisible to the > interpreter. And those frameworks are popular. You want > numpy.errstate/decimal.localcontext to work there too. > > 2. I'm curious how this will be implemented and what's > the performance impact. > > 3. I think that mechanism should be more generic than > just 'with' staments. What if you want to have a decorator > that applies some context? What if you can't write it as > a generator with @contextlib.contextmanager? > > What I would propose: > > I think that the right approach here would be to have a > standard protocol; something similar to how we defined > WSGI -- it doesn't require any changes in the interpreter, > it is a protocol. > > For instance, we could have a module in the stdlib, with a > class Context: > > class Context: > @classmethod > def __suspend_contexts(cls): > for child in cls.__subclasses__(): > child.__suspend_context() > > @classmethod > def __resume_contexts(cls): .. > > > To inform context objects that the context is about to change, > asyncio event loop would call Context.__suspend_contexts() > > I think that it's up to the event loop/library/framework > to manage context switches properly. In asyncio, for instance, > you can attach callbacks to Futures. I think it would be > great if such callbacks are executed in the right context. > > I would also suggest to think about a universal context -- > the one that works both for asyncio coroutines and > threadpools they might use. > > This way any framework can implement the context in the > most efficient way. > > All in all, I'm in favor of API and a couple functions > added to the stdlib for this. Great idea. I've been thinking of something along those lines. I think there also needs to be a recognition of active vs. inactive context. Only active contexts would be used by the event loop/etc. A generic context would also make it easy to clone and to use in context managers. One approach: class Context: def activate(self): ... # or open def deactivate(self): ... # or close def clone(self): ... def __suspend__(self, cs, id): ... def __resume__(self, cs, id): ... def __enter__(self): self.activate() return self def __exit__(self, ...): self.deactivate() def active(ctx=Context): return ... The various "context switchers" (event loop, thread pool, etc.) would then call __suspend__ and __resume__ on all the active contexts. The two args (cs for "context switcher") to those methods would allow each context object to adjust to the targeted "linear execution context" uniquely identified by the (cs, id) pair. For example: thread/async-local namespaces. I've often found it to be cleaner to split the "spec" from the "instance", so: class InactiveContext: # or GlobalContext or DefaultContext def activate(self): ... # return a new cloned Context def clone(self): ... # merge with activate? class Context: def deactivate(self): ... def clone(self): ... def __suspend__(self, cm, id): ... def __resume__(self, cm, id): ... def __enter__(self): return self def __exit__(self, ...): self.deactivate() This is all rough and the result of divided attention over the last few hours, but the idea of leaving the switching up to the "context switcher" is the key thing. Also, I smell some overlapping concepts with existing types and patterns that could probably distill the idea of a generic Context type into something cleaner. -eric From yselivanov.ml at gmail.com Thu Apr 30 06:17:54 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 30 Apr 2015 00:17:54 -0400 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: <554146C4.7010704@gmail.com> Message-ID: <5541ACF2.5070104@gmail.com> On 2015-04-29 11:28 PM, Guido van Rossum wrote: >> I would also suggest to think about a universal context -- >> >the one that works both for asyncio coroutines and >> >threadpools they might use. >> > >> >This way any framework can implement the context in the >> >most efficient way. >> > >> >All in all, I'm in favor of API and a couple functions >> >added to the stdlib for this. >> > > I'm curious -- how would you access the current contexts? I tried to find my implementation of context for asyncio (it was designed and tested to work with https://codereview.appspot.com/87940044/#ps1) but I couldn't. I'll need to spend some time to prototype this again. And not as a patch to asyncio, but as a generic thing. Yury From yselivanov.ml at gmail.com Thu Apr 30 06:26:13 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 30 Apr 2015 00:26:13 -0400 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: <554146C4.7010704@gmail.com> Message-ID: <5541AEE5.8040106@gmail.com> On 2015-04-29 8:53 PM, Nick Coghlan wrote: > * Can the same context be safely accessed from multiple threads > concurrently? At different points in time? In one of my context implementations (closed source) I monkey-patch threading module to actually track when threads are started to preserve the context information. Internally, the data storage is a tree, which branches with control points each time the context is modified. This way it's possible to see all data that was added to context "on the way" to the point that is currently executing by Python. > * Is there a "default context" which applies if no other context has > been explicitly activated? In all implementations I myself wrote, I always had a root context. But I don't like this design, especially for something generic in the standard library, where you want access to this root object to be namespaced (i.e. numpy cannot conflict with decimal by using the same key). I'm OK with some central hidden root context though, exposed only via an API for event loops, pools, etc, to facilitate the context switching. Thanks, Yury From njs at pobox.com Thu Apr 30 06:36:39 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 29 Apr 2015 21:36:39 -0700 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: Message-ID: On Wed, Apr 29, 2015 at 1:36 PM, Guido van Rossum wrote: > This seems reasonable, though mostly also non-urgent. Have you thought about > how it interacts with PEP 492 yet? The interaction with PEP 492 is pretty trivial AFAICT; they're almost entirely orthogonal. The only points of interaction are: 1) PEP 492 adds new suspension points beyond "yield" and "yield from" (i.e., "await" plus the implicit suspension points in "async for" and "async with"). This proposal tweaks the behavior of suspension points (so that they call __suspend__/__resume__). So obviously the solution is that if PEP 492 is accepted then we have to document that yes, the new suspension points also trigger __suspend__/__resume__ calls in the obvious way, just like "yield" does. 2) PEP 492 defines a new asynchronous context manager protocol, which is "the same as the regular context manager protocol, except with the letter 'a' added and they're coroutines instead of regular methods". This proposal adds stuff to the regular context manager protocol, so one would want to make the analogous changes to the asynchronous context manager protocol too. It's not 100% clear to me whether the __asuspend__/__aresume__ versions should be coroutines or not, but this is a pretty simple question to resolve. (I see downthread Yury that thinks not, so okay, I guess not, done :-).) So there aren't any real gotchas here AFAICT. -n > On Wed, Apr 29, 2015 at 1:22 PM, Nathaniel Smith wrote: >> >> Hi all, >> >> Well, after a few days no-one has responded to my post on another >> thread about this [1], but the more I thought about it the more this >> seemed like a good idea, so I wrote up a little more-formal proposal >> (attached) for letting context managers react to 'yield's that occur >> within their 'with' block. >> >> This should in many ways be treated as a straw man proposal -- there's >> tons I don't know about how async code is written in Python these days >> -- but it seems like a good idea to me and I'd like to hear what >> everyone else thinks :-). >> >> -n >> >> [1] https://mail.python.org/pipermail/python-ideas/2015-April/033176.html >> >> -- >> Nathaniel J. Smith -- http://vorpus.org >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) -- Nathaniel J. Smith -- http://vorpus.org From yselivanov.ml at gmail.com Thu Apr 30 06:39:20 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 30 Apr 2015 00:39:20 -0400 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: References: <554146C4.7010704@gmail.com> Message-ID: <5541B1F8.4000909@gmail.com> On 2015-04-30 12:04 AM, Eric Snow wrote: > This is all rough and the result of divided attention over the last > few hours, but the idea of leaving the switching up to the "context > switcher" is the key thing. Also, I smell some overlapping concepts > with existing types and patterns that could probably distill the idea > of a generic Context type into something cleaner. Agree. Long story short we have to prototype this ;) Yury From ericsnowcurrently at gmail.com Thu Apr 30 06:44:48 2015 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 29 Apr 2015 22:44:48 -0600 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: <5541B1F8.4000909@gmail.com> References: <554146C4.7010704@gmail.com> <5541B1F8.4000909@gmail.com> Message-ID: On Apr 29, 2015 10:39 PM, "Yury Selivanov" wrote: > Long story short we have to prototype this ;) Yep. :) -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Apr 30 08:27:05 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 29 Apr 2015 23:27:05 -0700 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: <554146C4.7010704@gmail.com> References: <554146C4.7010704@gmail.com> Message-ID: On Wed, Apr 29, 2015 at 2:01 PM, Yury Selivanov wrote: > Hi Nathanial, > > Sorry for not replying to your last email promptly. > FWIW I was going to do it later today ;) Oh well, this way you got a nice clean thread to do it in instead :-) > My opinion on this subject (and I've implemented lots > of local-context kind of objects in different frameworks) is > that *just* inserting some kind of suspend/resume points > before/after yields does not work. Before I get into the details of your message, I just want to emphasize-- there are clearly cases where inserting some kind of suspend/resume points before/after yields is not just a sufficient solution, but actually the only possible solution. Consider a streaming data processing pipeline that uses a generator to load data from disk in chunks, and then a series of generators to process, split, filter, etc. this data before passing it on to the next generator in the stack. For certain problems this is 100% the best way to do it, generators are amazing. But right now you kinda... just cannot use certain kinds of 'with' blocks in such code without all kinds of weirdness ensuing. And coming up with fancy APIs for event loops to use won't help, because there are no event loops here at all. The problem here isn't caused by event loops, it's caused by the way adding coroutines to Python created a dissociation between a block's dynamic-scope-in-terms-of-lifetime and its dynamic-scope-in-terms-of-call-chain. Obviously there may be other cases where you also need something else on top of this proposal (though I'm not 100% convinced... see below). But logically speaking, even if you're right that this proposal is not sufficient, that doesn't imply that it's not necessary :-). > Why: > > 1. Greenlets, gevent, eventlet, stackless python: they do > not have 'yield'. Context switches are invisible to the > interpreter. And those frameworks are popular. You want > numpy.errstate/decimal.localcontext to work there too. First, this is the only argument you're giving for "why" yield callbacks are insufficient, right? (I'm a little confused by your formatting and want to make sure I'm not missing anything.) Also, just to check, the only actual library that we really have to worry about here is greenlets, right, b/c gevent and eventlet use greenlets to do the heavy lifting, and stackless is a whole-interpreter fork so for them context switches are visible to the interpreter? Now, as to the actual point, I have a few reactions: a) The curmudgeonly reaction: Well, yes, the problem with those frameworks is that switches are invisible to both the interpreter and to user code, which makes it more difficult to manage shared state of all kinds, not just the very specific pseudo-dynamic-scoping state that we're talking about here. If you want a programming model where you can actually keep track of shared state, then maybe you should use something like asyncio instead :-). b) Okay, fine, if you want it to just work anyway: any implementation of this proposal will require that the interpreter have some way to walk the "with block stack" (so to speak) and call the __suspend__ and __resume__ methods on demand. greenlets could also walk up that stack when suspending execution. (This would require the library to do a bit of nasty mucking about in the CPython internals, but c'mon, this is greenlets we're talking about. Compared to the other stuff it does this would hardly even register.) The one caveat here is that if we want this to be possible then it would require we skip statically optimizing out __suspend__/__resume__ checking in regular functions. c) Backup plan: greenlets could provide an API like: greenlets.register_suspend_resume_callbacks(), greenlets.unregister_suspend_resume_callbacks(), and context managers that wanted to handle both kinds of suspensions could define __suspend__/__resume__ and then do def __enter__(self): greenlets.register_suspend_resume_callbacks(self.__suspend__, self.__resume__) # ... actual code ... def __exit__(self): greenlets.unregister_suspend_resume_callbacks(self.__suspend__, self.__resume__) # ... actual code ... and everything would AFAICT just work. Kinda annoying, but I'm pretty sure that there is and will always be exactly one greenlets library across the whole Python ecosystem, so it's not much of a limitation. > 2. I'm curious how this will be implemented and what's > the performance impact. I am also curious about the performance impact :-). I don't see any reason to expect it to be large, but one never knows for sure until one has an implementation. As for implementation, the most straightforward way I think would be to (1) add a variant of the SETUP_WITH bytecode that also looks up __suspend__ and __resume__ and pushes them into the value stack and then uses a unique b_type for its block stack entry, (2) add a variant of YIELD_VALUE that walks the block stack looking for this unique b_type and calls __suspend__ when found (b_level points to where to find it on the value stack), plus maybe as an optimization setting a flag on the frame noting that __resume__ will be needed, (3) tweak the resumption code so that it checks for this flag and if found walks the block stack calling __resume__, (4) add a variant of WITH_CLEANUP that knows how to clean up this more complex stack, (5) make sure that the unwind code knows to treat this new block type the same way it treats SETUP_FINALLY blocks. That's based on like 30 minutes of thought starting from zero knowledge of how this part of ceval.c works, so knowing ceval.c there are certainly a few dragons lurking that I've missed. But as an initial high-level sketch does it answer your question? > 3. I think that mechanism should be more generic than > just 'with' staments. What if you want to have a decorator > that applies some context? I don't understand the problem here. A coroutine decorator already knows when the wrapped coroutine is suspended/resumed, so there's nothing to fix -- basically for decorators this proposal is already implemented :-). > What if you can't write it as > a generator with @contextlib.contextmanager? Here I just don't understand the sentence :-). Can you rephrase? Answering one possible question you might have intended: @contextmanager indeed is not useful for defining a context manager that has __suspend__ and __resume__ callbacks, but this is just a convenience utility. My suspicion is that these kinds of context managers are already sufficiently rare and tricky to write that having special utilities to make it easier is not that important, but if someone comes up with a nice API for doing so then cool, we could put that in contextlib. > What I would propose: > > I think that the right approach here would be to have a > standard protocol; something similar to how we defined > WSGI -- it doesn't require any changes in the interpreter, > it is a protocol. > > For instance, we could have a module in the stdlib, with a > class Context: > > class Context: > @classmethod > def __suspend_contexts(cls): > for child in cls.__subclasses__(): > child.__suspend_context() > > @classmethod > def __resume_contexts(cls): .. > > > To inform context objects that the context is about to change, > asyncio event loop would call Context.__suspend_contexts() I can't see how to usefully implement, well, anything, using this interface as given. Maybe you mean something like class Context: @classmethod def switch_contexts(cls, old_ctx_id, new_ctx_id): for child in cls.__subclasses__(): child.switch_contexts(old_ctx_id, new_ctx_id) @classmethod def destroy_context(cls, ctx_id): for child in cls.__subclasses__(): child.destroy_context(ctx_id) ? Or a better approach that avoids having to generate and keep track of arbitrary identifiers would be: class Context: @classmethod def new_context(cls): return {subcls: subcls.new_context() for subcls in cls.__subclasses__()} @classmethod def get_context(cls): return {subcls: subcls.get_context() for subcls in cls.__subclasses__()} @classmethod def set_context(cls, context): for subcls in cls.__subclasses__(): subcls.set_context(context[subcls]) And then you could use it like: class myframework_context: def __enter__(self): self.old_context = Context.get_context() self.context = Context.new_context() Context.set_context(self.context) def __suspend__(self): Context.set_context(self.old_context) def __resume__(self): Context.set_context(self.context) def __exit__(self): Context.set_context(self.old_context) def myframework_run_task(coro): with myframework_context: yield from coro and voila, works even if you have multiple event loops calling into each other :-) Okay, I kid, but my point is that these different approaches really are not very different. The big advantage of this approach over ones that involve doing stuff like main_loop.get_dynamic_namespace() is that it lets each piece of state be stored for regular usage in whatever form makes sense -- e.g. numpy's errstate can live in a plain old TLS variable and be accessed quickly from there, and we only have to do expensive stuff at context switch time. But this also means that your context switches are now notionally identical to my context switches -- they call some suspend/resume callbacks and let each piece of state push/pop itself as appropriate. The real difference is just that you assume that every Python program has exactly one piece of manager code somewhere that knows about all context switches, so you have it do the calls, and I assume that there may be many context switch points within a single program (because, well, Python has dedicated syntax for such context switches built into the language itself), so I think Python itself should do the calls at each appropriate place. > I think that it's up to the event loop/library/framework > to manage context switches properly. In asyncio, for instance, > you can attach callbacks to Futures. I think it would be > great if such callbacks are executed in the right context. Again, these approaches aren't necessarily contradictory. For something like a request id to be used in logging, which is tightly coupled to the flow of data through your async application, what you say makes perfect sense. For something like the decimal context, it feels pretty weird to me. > I would also suggest to think about a universal context -- > the one that works both for asyncio coroutines and > threadpools they might use. I'm afraid I have no idea what this means either. -n -- Nathaniel J. Smith -- http://vorpus.org From njs at pobox.com Thu Apr 30 08:31:45 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 29 Apr 2015 23:31:45 -0700 Subject: [Python-ideas] Letting context managers react to yields inside their scope In-Reply-To: <55414D26.7020602@gmail.com> References: <55414D26.7020602@gmail.com> Message-ID: On Wed, Apr 29, 2015 at 2:29 PM, Yury Selivanov wrote: > Nathanial, > > On 2015-04-29 4:22 PM, Nathaniel Smith wrote: >> >> Interaction with PEP 492 >> ======================== >> >> PEP 492 currently proposes an ``async with`` statement. Despite the >> name, this is actually orthogonal to the proposal here: this PEP >> proposes to allow context managers to react to suspensions inside >> their scope, while ``async with`` makes context manager ``__enter__`` >> and ``__exit__`` methods into potential ``yield`` points. If both >> PEPs are accepted then we will presumably also want to add some kind >> of ``__asuspend__`` and ``__aresume__`` methods to PEP 492-style >> asynchronous context managers. > > I doubt that we'll need __asuspend__. There shouldn't be > a need to perform any blocking operations there. > > __suspend__ and __resume__ would be enough. I couldn't think of any cases either, but said __asuspend__/__aresume__ to play it safe in case I missed any :-). I guess it makes sense, though, that from outside our thread, suspend/resume operations are invisible -- no-one cares whether the coroutine they're talking to is currently scheduled or currently suspended, that's purely internal. Therefore, suspend/resume callbacks should never need to block, yeah. -n -- Nathaniel J. Smith -- http://vorpus.org From toddrjen at gmail.com Thu Apr 30 11:48:21 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 30 Apr 2015 11:48:21 +0200 Subject: [Python-ideas] More general "for" loop handling Message-ID: Looking at pep 492, it seems to me the handling of "for" loops has use outside of just asyncio. The primary use-case I can think of is multiprocessing and multithreading. For example, you could create a multiprocessing pool, and let the pool handle the items in a "for" loop, like so: from multiprocessing import Pool mypool = Pool(10, maxtasksperchild=2) mypool for item in items: do_something_here do_something_else do_yet_another_thing Or something similar with third-party modules: from greenlet import greenlet greenlet for item in items: do_something_here do_something_else do_yet_another_thing Of course this sort of thing is possible with iterators and maps today, but I think a lot of the same advantages that apply to asyncio also apply to these sorts of cases. So I think that, rather than having a special keyword just for asyncio, I think it would be better to have a more flexible approach. Perhaps something like a "__for__" magic method that lets a class implement "for" loop handling, along with the corresponding changes in how the language processes the "for" loop. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Apr 30 12:00:28 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Apr 2015 11:00:28 +0100 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: Message-ID: On 30 April 2015 at 10:48, Todd wrote: > Of course this sort of thing is possible with iterators and maps today, but > I think a lot of the same advantages that apply to asyncio also apply to > these sorts of cases. So I think that, rather than having a special keyword > just for asyncio, I think it would be better to have a more flexible > approach. Perhaps something like a "__for__" magic method that lets a class > implement "for" loop handling, along with the corresponding changes in how > the language processes the "for" loop. +1 on making a more general construct than "async for", which can then be used to implement an equivalent to "async for" as well as similar constructs for threads processes and whatever else 3rd party code might find a use for. Paul From steve at pearwood.info Thu Apr 30 13:36:45 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Apr 2015 21:36:45 +1000 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: Message-ID: <20150430113644.GC5663@ando.pearwood.info> On Thu, Apr 30, 2015 at 11:48:21AM +0200, Todd wrote: > Looking at pep 492, it seems to me the handling of "for" loops has use > outside of just asyncio. The primary use-case I can think of is > multiprocessing and multithreading. > > For example, you could create a multiprocessing pool, and let the pool > handle the items in a "for" loop, like so: > > from multiprocessing import Pool > > mypool = Pool(10, maxtasksperchild=2) > > mypool for item in items: > do_something_here > do_something_else > do_yet_another_thing That's a very pretty piece of pseudo-code (actually, I lie, I don't think it is pretty at all, but for the sake of the argument let's pretend it is) but what does it do? How does it do it? Let's be concrete: mypool = Pool(10, maxtasksperchild=2) items = range(1000) mypool for item in items: print(item) if item == 30: break x = item + 1 print(x) What gets printed? A parallel version of map makes sense, because the semantics of map are well defined: given a function f and a sequence [a, b, c, ...] it creates a new sequence [f(a), f(b), f(c), ...]. The assumption is that f is a pure-function which is side-effect free (if it isn't, you're going to have a bad time). The specific order in which a, b, c etc. are processed doesn't matter. If it does matter, then map is the wrong way to process it. But a parallel version of for does not make sense to me. (I must admit, I'm having trouble understanding what the "async for" will do too.) By definition, a for-loop is supposed to be sequential. Loop the first time, *then* the second time, *then* the third time. There's no presumption of the body of the for-block being side-effect free, and you're certainly not free to perform the loops in some other order. > Of course this sort of thing is possible with iterators and maps today, but > I think a lot of the same advantages that apply to asyncio also apply to > these sorts of cases. So I think that, rather than having a special > keyword just for asyncio, I think it would be better to have a more > flexible approach. Perhaps something like a "__for__" magic method that > lets a class implement "for" loop handling, along with the corresponding > changes in how the language processes the "for" loop. "async for" hasn't proven itself yet, and you are already looking to generalise it? Shouldn't it prove itself as not a mistake first? -- Steve From stefan_ml at behnel.de Thu Apr 30 18:03:55 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 30 Apr 2015 18:03:55 +0200 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: <20150430113644.GC5663@ando.pearwood.info> References: <20150430113644.GC5663@ando.pearwood.info> Message-ID: Steven D'Aprano schrieb am 30.04.2015 um 13:36: > "async for" hasn't proven itself yet, and you are already looking to > generalise it? Shouldn't it prove itself as not a mistake first? Also, it should be quite possible to achieve what the OP proposed with "async for" since it's in no way limited to the way asyncio handles things. "async for" is a bit of a badly named feature, but that's intended in order to match what people would know from other programming languages. Stefan From p.f.moore at gmail.com Thu Apr 30 18:45:56 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Apr 2015 17:45:56 +0100 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: <20150430113644.GC5663@ando.pearwood.info> Message-ID: On 30 April 2015 at 17:03, Stefan Behnel wrote: > Steven D'Aprano schrieb am 30.04.2015 um 13:36: >> "async for" hasn't proven itself yet, and you are already looking to >> generalise it? Shouldn't it prove itself as not a mistake first? > > Also, it should be quite possible to achieve what the OP proposed with > "async for" since it's in no way limited to the way asyncio handles things. > "async for" is a bit of a badly named feature, but that's intended in order > to match what people would know from other programming languages. Could you explain how? Specifically, what's the translation of from multiprocessing import Pool mypool = Pool(10, maxtasksperchild=2) mypool for item in items: do_something_here do_something_else do_yet_another_thing I'm assuming that's the OP's intention (it's certainly mine) is that the "mypool for" loop works something like def _work(item): do_something_here do_something_else do_yet_another_thing for _ in mypool.map(_work, items): # Wait for the subprocesses pass How would I use "async for" to get the same result? (And the same for a concurrent.futures Executor in place of a multiprocessing pool). Paul. From yselivanov.ml at gmail.com Thu Apr 30 18:58:41 2015 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 30 Apr 2015 12:58:41 -0400 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: Message-ID: <55425F41.2060000@gmail.com> On 2015-04-30 5:48 AM, Todd wrote: > For example, you could create a multiprocessing pool, and let the pool > handle the items in a "for" loop, like so: > > from multiprocessing import Pool > > mypool = Pool(10, maxtasksperchild=2) > > mypool for item in items: This looks "OK" for a simple snippet, but how will you define this new syntax in Python grammar? Unless you restrict such syntax to use only NAME tokens before 'for', you can easily expect users to write code like this: some_namespace.module.function(arg=123) for item in items(): ... Yury From toddrjen at gmail.com Thu Apr 30 19:07:34 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 30 Apr 2015 19:07:34 +0200 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: <55425F41.2060000@gmail.com> References: <55425F41.2060000@gmail.com> Message-ID: On Thu, Apr 30, 2015 at 6:58 PM, Yury Selivanov wrote: > > > On 2015-04-30 5:48 AM, Todd wrote: > >> For example, you could create a multiprocessing pool, and let the pool >> handle the items in a "for" loop, like so: >> >> from multiprocessing import Pool >> >> mypool = Pool(10, maxtasksperchild=2) >> >> mypool for item in items: >> > > > This looks "OK" for a simple snippet, but how will you define > this new syntax in Python grammar? > > Unless you restrict such syntax to use only NAME tokens before > 'for', you can easily expect users to write code like this: > > some_namespace.module.function(arg=123) for item in items(): > ... > pep8, probably. You can write ugly code now. This isn't much better: some_namespace.module.function(arg=123).map(items()) -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Apr 30 19:12:11 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 30 Apr 2015 19:12:11 +0200 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: <20150430113644.GC5663@ando.pearwood.info> References: <20150430113644.GC5663@ando.pearwood.info> Message-ID: On Thu, Apr 30, 2015 at 1:36 PM, Steven D'Aprano wrote: > On Thu, Apr 30, 2015 at 11:48:21AM +0200, Todd wrote: > > Looking at pep 492, it seems to me the handling of "for" loops has use > > outside of just asyncio. The primary use-case I can think of is > > multiprocessing and multithreading. > > > > For example, you could create a multiprocessing pool, and let the pool > > handle the items in a "for" loop, like so: > > > > from multiprocessing import Pool > > > > mypool = Pool(10, maxtasksperchild=2) > > > > mypool for item in items: > > do_something_here > > do_something_else > > do_yet_another_thing > > > A parallel version of map makes sense, because the semantics of map are > well defined: given a function f and a sequence [a, b, c, ...] it > creates a new sequence [f(a), f(b), f(c), ...]. The assumption is that f > is a pure-function which is side-effect free (if it isn't, you're going > to have a bad time). The specific order in which a, b, c etc. are > processed doesn't matter. If it does matter, then map is the wrong way > to process it. > > multiprocessing.Pool.map guarantees ordering. It is multiprocessing.Pool.imap_unordered that doesn't. > > Of course this sort of thing is possible with iterators and maps today, > but > > I think a lot of the same advantages that apply to asyncio also apply to > > these sorts of cases. So I think that, rather than having a special > > keyword just for asyncio, I think it would be better to have a more > > flexible approach. Perhaps something like a "__for__" magic method that > > lets a class implement "for" loop handling, along with the corresponding > > changes in how the language processes the "for" loop. > > "async for" hasn't proven itself yet, and you are already looking to > generalise it? Shouldn't it prove itself as not a mistake first? > > Two reasons: 1. It may be hard to generalize it later without breaking backwards compatibility. 2. Whether it can be generalized or not may have some bearing on whether it gets accepted. -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Thu Apr 30 19:13:37 2015 From: toddrjen at gmail.com (Todd) Date: Thu, 30 Apr 2015 19:13:37 +0200 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: <20150430113644.GC5663@ando.pearwood.info> Message-ID: On Thu, Apr 30, 2015 at 6:45 PM, Paul Moore wrote: > On 30 April 2015 at 17:03, Stefan Behnel wrote: > > Steven D'Aprano schrieb am 30.04.2015 um 13:36: > >> "async for" hasn't proven itself yet, and you are already looking to > >> generalise it? Shouldn't it prove itself as not a mistake first? > > > > Also, it should be quite possible to achieve what the OP proposed with > > "async for" since it's in no way limited to the way asyncio handles > things. > > "async for" is a bit of a badly named feature, but that's intended in > order > > to match what people would know from other programming languages. > > Could you explain how? > > Specifically, what's the translation of > > from multiprocessing import Pool > > mypool = Pool(10, maxtasksperchild=2) > > mypool for item in items: > do_something_here > do_something_else > do_yet_another_thing > > I'm assuming that's the OP's intention (it's certainly mine) is that > the "mypool for" loop works something like > > def _work(item): > do_something_here > do_something_else > do_yet_another_thing > for _ in mypool.map(_work, items): > # Wait for the subprocesses > pass > Yes, thank you, that is exactly what I intended. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Apr 30 19:31:40 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 30 Apr 2015 10:31:40 -0700 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: <20150430113644.GC5663@ando.pearwood.info> Message-ID: Ah. But 'async for' is not meant to introduce parallelism or concurrency. It is only meant to be able to insert specific places during a sequential iteration where a coroutine's stack can be suspended. The primitives proposed by PEP 492 don't introduce new ways to spell concurrency -- for that you would need things like asyncio.gather(). If you want to introduce ways to spell concurrency directly in the language you'll have to write and defend your own PEP. PEP 492 is only meant to make code easier to read and write that's already written to use coroutines (e.g. using the asyncio library, but not limited to that). On Thu, Apr 30, 2015 at 10:13 AM, Todd wrote: > > > On Thu, Apr 30, 2015 at 6:45 PM, Paul Moore wrote: > >> On 30 April 2015 at 17:03, Stefan Behnel wrote: >> > Steven D'Aprano schrieb am 30.04.2015 um 13:36: >> >> "async for" hasn't proven itself yet, and you are already looking to >> >> generalise it? Shouldn't it prove itself as not a mistake first? >> > >> > Also, it should be quite possible to achieve what the OP proposed with >> > "async for" since it's in no way limited to the way asyncio handles >> things. >> > "async for" is a bit of a badly named feature, but that's intended in >> order >> > to match what people would know from other programming languages. >> >> Could you explain how? >> >> Specifically, what's the translation of >> >> from multiprocessing import Pool >> >> mypool = Pool(10, maxtasksperchild=2) >> >> mypool for item in items: >> do_something_here >> do_something_else >> do_yet_another_thing >> >> I'm assuming that's the OP's intention (it's certainly mine) is that >> the "mypool for" loop works something like >> >> def _work(item): >> do_something_here >> do_something_else >> do_yet_another_thing >> for _ in mypool.map(_work, items): >> # Wait for the subprocesses >> pass >> > > Yes, thank you, that is exactly what I intended. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Apr 30 19:54:50 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Apr 2015 18:54:50 +0100 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: <20150430113644.GC5663@ando.pearwood.info> Message-ID: On 30 April 2015 at 18:31, Guido van Rossum wrote: > PEP 492 is only meant to make code easier to read and write that's already > written to use coroutines (e.g. using the asyncio library, but not limited > to that). OK, that's fair. To an outsider like me it feels like a lot of new syntax to support a very specific use case. But that's because I don't really have a feel for what you mean when you note "but not limited to that". Are there any good examples or use cases for coroutines that are *not* asyncio-based? And assuming you are saying that PEP 482 should help for those as well, could it include a non-asyncio example? My immediate reaction is that the keywords "async" and "await" will seem a little odd in a non-asyncio context. Paul Paul From guido at python.org Thu Apr 30 20:18:16 2015 From: guido at python.org (Guido van Rossum) Date: Thu, 30 Apr 2015 11:18:16 -0700 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: <20150430113644.GC5663@ando.pearwood.info> Message-ID: On Thu, Apr 30, 2015 at 10:54 AM, Paul Moore wrote: > On 30 April 2015 at 18:31, Guido van Rossum wrote: > > PEP 492 is only meant to make code easier to read and write that's > already > > written to use coroutines (e.g. using the asyncio library, but not > limited > > to that). > > OK, that's fair. To an outsider like me it feels like a lot of new > syntax to support a very specific use case. But that's because I don't > really have a feel for what you mean when you note "but not limited to > that". Are there any good examples or use cases for coroutines that > are *not* asyncio-based? And assuming you are saying that PEP 482 > should help for those as well, could it include a non-asyncio example? > My immediate reaction is that the keywords "async" and "await" will > seem a little odd in a non-asyncio context. > The async/await pair of keywords for coroutines actually stems from C#, where the compiler generates special coroutine suspension code when it sees them, and the type checker verifies that they are used correctly -- await's argument must be something of type async, and await must occur inside a function declared as async. These semantics are very similar to coroutines using yield-from in Python, and I had observed the similarity long before this PEP was written. Most examples of coroutines will be doing some kind of I/O multiplexing, because that's what they're good for. But asyncio is not the only explicit I/O multiplexing system in the Python world. Twisted had yield-based coroutines long before anyone else (they called them InlineCallbacks though) and is likely to support Python 3 and some level of interoperability with asyncio. Note: it's PEP 492, not 482. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Thu Apr 30 20:31:04 2015 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 30 Apr 2015 15:31:04 -0300 Subject: [Python-ideas] More general "for" loop handling In-Reply-To: References: Message-ID: On 30 April 2015 at 06:48, Todd wrote: > Looking at pep 492, it seems to me the handling of "for" loops has use > outside of just asyncio. The primary use-case I can think of is > multiprocessing and multithreading. > > For example, you could create a multiprocessing pool, and let the pool > handle the items in a "for" loop, like so: > > from multiprocessing import Pool > > mypool = Pool(10, maxtasksperchild=2) > > mypool for item in items: > do_something_here > do_something_else > do_yet_another_thing While the idea is cool, maybe the original "for" is quite enough for doing that - as can be seen in this real-world (if simple) package: https://github.com/npryce/python-parallelize --------------- import os from parallelize import parallelize for i in parallelize(range(100)): print(os.getpid(), i) From oscar.j.benjamin at gmail.com Thu Apr 30 22:37:51 2015 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 30 Apr 2015 21:37:51 +0100 Subject: [Python-ideas] PEP 492 terminology - (native) coroutine objects Message-ID: Looking through PEP 492 I dislike the terminology. With generators I can do: >>> def f(): yield ... >>> g = f() >>> g So f is a generator function and g is generator. That's all pretty clear and "generator" is IMO a great name. The term "generator function" is also unambiguous for f. With PEP 492 it seems that I would get something like: >>> async def af(): pass >>> ag = af() >>> ag According to the terminology in the PEP af is a coroutine but since the word coroutine also refers to some generators in a looser sense I should use "native coroutine" to disambiguate. ag is a "coroutine object" but again I should use "native coroutine object" to explicitly name the type because without the word "native" it also refers to other things. I think that if the terminology is already ambiguous with related language constructs before it's introduced then it would be better to change it now. The word coroutine (without "native" modifier) is used by the PEP to refer to both generator based coroutines and the new async def functions. I think it's reasonable to use the word as a generic term in this sense. Python's idea of coroutines (both types) doesn't seem to match with the pre-existing general definitions but they are at least generalisations of functions so it can be reasonable to describe them that way loosely. Greg's suggestion to call an async def function (af above) an "async function" seems like a big improvement. It clearly labels the purpose: a function for use in a asynchronous execution probably with the asyncio module. It also matches directly with the syntax: a function prefixed with the word "async". There would be no ambiguity between which of af or ag is referred to by the term. It seems harder to think of a good name for ag though. ag is a wrapper around a suspended call stack with methods to resume the stack so describing what is is doesn't lead to anything helpful. OTOH the purpose of ag is often described as implementing a "minithread" so the word "minithread" makes intuitive sense to me. That way async code is done by writing async functions that return minithreads. An event loop schedules the minthreads etc. (I'm trying to imagine explaining this to someone...) I'm not sure what the best word is for a coroutine object but the current terminology clearly has room for improvement. For a start using the word "object" in the name of a type is a bit rubbish in a language where everything is an object. Worse the PEP is reusing words that have already been used with different meanings so that it's already ambiguous. A concrete builtin language type such as the coroutine object deserves a good, short, one-word name that will not be confused with other things. -- Oscar