From njs at pobox.com Tue Jul 5 20:56:50 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 5 Jul 2016 17:56:50 -0700 Subject: [Async-sig] Asynchronous cleanup is a problem Message-ID: So here's an interesting issue I just discovered while experimenting with async generators. It caught me by surprise, anyway. Maybe this was already obvious to everyone else. But I wanted to get some more perspectives. Starting axiom: async functions / async generators need to be prepared for the case where they get garbage collected before being run to completion, because, well... this is a thing that can happen. Currently, the way garbage collection is handled is that their __del__ method calls their .close() method, which does something like: class GeneratorType: ... def close(self): try: self.throw(GeneratorExit) except GeneratorExit, StopIteration: return # it worked, all is good except: raise # double-fault, propagate else: raise RuntimeError("generator ignored GeneratorExit") (see PEP 342). So far, so obvious -- an async function that gets a GeneratorExit has to propagate that exception immediately. This is a regular method, not an async method, because it Dismaying conclusion: inside an async function / async generator, finally: blocks must never yield (and neither can __aexit__ callbacks), because they might be invoked during garbage collection. For async functions this is... arguably a problem but not super urgent, because async functions rarely get garbage collected without being run to completion. For async generators it's a much bigger problem. There's some more discussion, and a first sketch at conventions we might want to use for handling this, here: https://github.com/dabeaz/curio/issues/70 -n -- Nathaniel J. Smith -- https://vorpus.org From cory at lukasa.co.uk Wed Jul 6 05:00:49 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 6 Jul 2016 10:00:49 +0100 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: Message-ID: <6AEAC6BB-9807-4217-9B9F-19A87ABC9147@lukasa.co.uk> > On 6 Jul 2016, at 01:56, Nathaniel Smith wrote: > > There's some more discussion, and a first sketch at conventions we > might want to use for handling this, here: > > https://github.com/dabeaz/curio/issues/70 This feels like a problem with very few good solutions. Finalizers (e.g. __del__ and weakref callbacks) obviously cannot await for anything (they have no coroutine runner to yield to, and there is currently no Python API for handing off to a coroutine runner to execute your finalizer). That strongly suggests that all cleanup inside coroutines and async generators must be synchronous. This is especially problematic given the existence of ?async with?, which nominally promises to do asynchronous cleanup. What?s not entirely clear to me is why we need __aexit__ to *actually* be an async function. The example in curio is socket closure, which seems to be like it absolutely does not need to be awaitable. Why can?t close() just tell the event loop (or curio kernel) that I?m done with the socket, and to clean up that socket on its own time (again, this is what Twisted does). In the worst case you could rule that __aexit__ cannot use coroutines, but of course it may continue to use Futures and other fun things. I know that callbacks aren?t The Asyncio Way, but they have their uses, and this is probably one of them. Allowing Futures/Deferreds allows your __aexit__ to do some actual I/O if that?s required (e.g. send a HTTP/2 GOAWAY frame). The only reason I can think of that __aexit__ needs to be a coroutine is to guarantee that the resource in question is genuinely cleaned up by the time the with block is exited. It is not entirely clear to me what the value of this guarantee is: does anyone have a good use-case for it that doesn?t seem like it violates the spirit of the context manager? That leaves us with finally, and here I have no good solution except that finally inside generators has *always* been a problem. However, in this case, I think we?ve got a good analogy. In synchronous generators, if you yield inside a finally *and* leak your coroutine, the interpreter moans at you. I don?t see any reason not to treat await/yield from inside finally in exactly the same way: if you do it, you eventually explode. Basically, there doesn?t seem to be an obvious way to make garbage collection of coroutines work while allowing them to be coroutines without having some way to register a coroutine runner with the garbage collector. That strikes me as a *terrible* idea (e.g. you may have multiple event loops, which requires that you register a unique coroutine runner per coroutine). Therefore, the only logical thing to do is to have only synchronous functions invoked in cleanup methods (e.g. __aexit__ and finally), and if those need to do some form of asynchronous I/O they need to use a Future-like construct to actually achieve it. Cory -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From dave at dabeaz.com Wed Jul 6 08:09:04 2016 From: dave at dabeaz.com (David Beazley) Date: Wed, 6 Jul 2016 07:09:04 -0500 Subject: [Async-sig] Asynchronous cleanup is a problem Message-ID: Cory wrote: "What?s not entirely clear to me is why we need __aexit__ to *actually* be an async function. The example in curio is socket closure, which seems to be like it absolutely does not need to be awaitable." Curio uses asynchronous context managers for much more than closing sockets (which frankly is the least interesting thing). For example, they're used extensively with synchronization primitives such as Locks, Semaphores, Events, Queues, and other such things. The ability to use coroutines in the __aexit__() method is an essential part of these primitives because it allows task scheduling decisions to be made in conjunction with synchronization events such as lock releases. For example, you can implement fair-locking or various forms of priority scheduling. Curio also uses asynchronous context managers for timeouts and other related functionality where coroutines have to be used in __aexit__. I would expect coroutines in __aexit__ to also be useful in more advanced contexts such as working with databases, dealing with transactions, and other kinds of processing where asynchronous I/O might be involved. As such, I think it's pretty useful to allow coroutines in __aexit__() and would be strongly opposed to restricting it. -Dave From cory at lukasa.co.uk Wed Jul 6 09:42:27 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 6 Jul 2016 14:42:27 +0100 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: Message-ID: > On 6 Jul 2016, at 13:09, David Beazley wrote: > > Curio uses asynchronous context managers for much more than closing sockets (which frankly is the least interesting thing). For example, they're used extensively with synchronization primitives such as Locks, Semaphores, Events, Queues, and other such things. The ability to use coroutines in the __aexit__() method is an essential part of these primitives because it allows task scheduling decisions to be made in conjunction with synchronization events such as lock releases. For example, you can implement fair-locking or various forms of priority scheduling. Curio also uses asynchronous context managers for timeouts and other related functionality where coroutines have to be used in __aexit__. I would expect coroutines in __aexit__ to also be useful in more advanced contexts such as working with databases, dealing with transactions, and other kinds of processing where asynchronous I/O might be involved. For my own edification, Dave, do you mind if I dive into this a little bit? My suspicion is that this problem is rather unique to curio (or, at least, does not so strongly effect event loop implementations), and I?d just like to get a handle on it. It?s important to me that we don?t blow away curio, so where it differs from event loops I?d like to understand what it?s doing. In the case of an event loop implementation, all of the above can be implemented simply by scheduling a callback which can then do whatever it needs to do. For example, scheduling a fair lock can be implemented synchronously, either literally or by means of something like asyncio.call_soon(). The only advantage I can see for event loops in being able to use a coroutine here is that they can suspend execution of the outer block until such time as they *know* that the next task has been scheduled into the critical section, which while useful does not strike me as *necessary*. However, as far as I know curio doesn?t have the ability to schedule an operation in a synchronous manner by means of something like a Future. Is that correct? If there is no way in curio to spawn a task to occur later without having to await on it, then clearly there is no option but to allow coroutines in __aexit__ and finally: how else could curio operate? Cory -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From yselivanov at gmail.com Wed Jul 6 11:50:32 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 6 Jul 2016 11:50:32 -0400 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: Message-ID: <5DE79CA2-D30A-4728-8953-F165A8DA1EAB@gmail.com> > On Jul 5, 2016, at 8:56 PM, Nathaniel Smith wrote: [..] > Starting axiom: async functions / async generators need to be prepared > for the case where they get garbage collected before being run to > completion, because, well... this is a thing that can happen. [..] > Dismaying conclusion: inside an async function / async generator, > finally: blocks must never yield (and neither can __aexit__ > callbacks), because they might be invoked during garbage collection. I agree with David here: coroutines are always running under a scheduler which, in one way or another, keeps strong references to them. In curio it?s a global table of all tasks, in asyncio it?s a chain of references to callbacks of Tasks/Futures. The only time a coroutine can be GC?ed in asyncio is when the caller did not use ?await? on it. A running coroutine being GC?ed is an exceptional situation, that means that there is a bug in your scheduler. And people actually rely on this thing. It?s not so much about __aexit__ returning an awaitable, it?s about people writing code that awaits in ?finally? statements. That?s why I?m big -1 on changing __aexit__. If we change __aexit__, we should also prohibit awaiting in finally blocks, which is not an option. > For async functions this is... arguably a problem but not super > urgent, because async functions rarely get garbage collected without > being run to completion. For async generators it's a much bigger > problem. I?m not sure I understand why it?d a problem for async generators. Since coroutines shouldn?t ever be GC?ed while running, async generators generally won?t be GC?ed while running too, because they will have a strong ref from the running coroutine. I think to makes things simple, we shouldn?t have a ?close()? method on async generators at all. When a running async generator is GC?ed we?ll make the interpreter to issue a warning. We might want to add an ?aclose()? coroutine method, which will throw a GeneratorExit exception (or GeneratorAsyncExit) with the following semantics: 1. If the running async generator ignores GeneratorAsyncExit and keeps ?async yielding?, we throw a RuntimeError. 2. If the running async generator receives a GeneratorAsyncExit exception outside of a try..finally block, the async generator is closed silently. 3. If the running async generator receives a GeneratorAsyncExit exception inside a ?finally? block, it will be able to await on any number of coroutines inside that block. Thanks, Yury From glyph at twistedmatrix.com Wed Jul 6 15:46:56 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 6 Jul 2016 12:46:56 -0700 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: Message-ID: > On Jul 5, 2016, at 5:56 PM, Nathaniel Smith wrote: > > Starting axiom: async functions / async generators need to be prepared > for the case where they get garbage collected before being run to > completion, because, well... this is a thing that can happen. This thread is really confusing, so I'm going to try to just attack this axiom (which isn't really axiomatic, it's a conclusion drawn from a few other properties of the interpreter and the event loop) and see if it holds :-). Yury already pointed out that coroutines run under a scheduler that keeps strong references to them. Backing that up a little bit - by definition, if your coroutine is not strongly referenced by a scheduler, it can't be doing anything interesting; nobody's going to call .send on it. This is why 'foo()' is a no-op whereas 'await foo()' actually causes something to happen. Similarly, this is why you need Task() to do something in parallel, and you can't just toss a coroutine out into the void. Furthermore, Task.cancel()'s documentation holds another clue about asynchronous cleanup: "Unlike Future.cancel(), this does not guarantee that the task will be cancelled: the exception might be caught and acted upon, delaying cancellation of the task or preventing cancellation completely". Deferred.cancel() behaves in much the same way, for the same reason. It is explicitly allowed that an asynchronous task do asynchronous clean-up. Now, if you have a task that was scheduled but has come forcibly detached from its scheduler, you are in a bit of a mess. But this is an inherently unresolvable mess, for the same reason that Thread.kill() is an inherently unresolvable mess: you cannot forcibly terminate a thread of execution and end up with your program in a known state. It's OK to kill a generator (as distinct from a 'coroutine' in the sense that it does not expect .send to be called on it) from the GC because a generator is just yielding values and since it doesn't expect .send it can't expect to keep running, and it can do purely synchronous try/finally. But as soon as you are allowed to invoke asynchronous work, you have to be allowed to let that asynchronous work complete. So: async functions should not and cannot be prepared for the case where they get garbage collected; this is not a thing that can happen unless the coroutine scheduler is damaged beyond repair and it's time to crash your process. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jul 6 15:54:47 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jul 2016 12:54:47 -0700 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: Message-ID: On Wed, Jul 6, 2016 at 6:42 AM, Cory Benfield wrote: > >> On 6 Jul 2016, at 13:09, David Beazley wrote: >> >> Curio uses asynchronous context managers for much more than closing sockets (which frankly is the least interesting thing). For example, they're used extensively with synchronization primitives such as Locks, Semaphores, Events, Queues, and other such things. The ability to use coroutines in the __aexit__() method is an essential part of these primitives because it allows task scheduling decisions to be made in conjunction with synchronization events such as lock releases. For example, you can implement fair-locking or various forms of priority scheduling. Curio also uses asynchronous context managers for timeouts and other related functionality where coroutines have to be used in __aexit__. I would expect coroutines in __aexit__ to also be useful in more advanced contexts such as working with databases, dealing with transactions, and other kinds of processing where asynchronous I/O might be involved. > > For my own edification, Dave, do you mind if I dive into this a little bit? My suspicion is that this problem is rather unique to curio (or, at least, does not so strongly effect event loop implementations), and I?d just like to get a handle on it. It?s important to me that we don?t blow away curio, so where it differs from event loops I?d like to understand what it?s doing. The relevant difference between curio and asyncio here is that in asyncio, there are two different mechanisms for accessing the event loop: for some operations, you access it through its coroutine runner interface using 'await', and for other operations, you get a direct reference to it through some side-channel (loop= arguments, global lookups) and then make direct method calls. To make the latter work, asyncio code generally has to be written in a way that makes sure loop= objects are always passed throughout the whole call stack, and always stay in sync [no pun intended] with the coroutine runner object. This has a number of downsides, but one upside is that it means that the loop object is available from __del__, where the coroutine runner isn't. Curio takes the other approach, of standardizing on 'await' as the single blessed mechanism for accessing the event loop (or kernel or whatever you want to call it). So this eliminates all the tiresome loop= tracking and the potential for out-of-sync bugs, but it means you can't do *anything* event-loop-related from __del__. However, I'm not convinced that asyncio really has an advantage here. Imagine some code that uses asyncio internally, but exposes a synchronous wrapper to the outside (e.g. as proposed here ;-) [1]): def synchronous_wrapper(...): # Use a new loop since we might not be in the same thread as the global loop loop = asyncio.new_event_loop() return loop.run_until_complete(work_asynchronously(..., loop=loop)) async def work_asynchronously(..., loop=loop): stream = await get_asyncio_stream_writer(..., loop=loop) stream.write(b"hello") stream.write(...) queues some data to be sent, but doesn't send it. Then stream falls out of scope, which triggers a call to stream.__del__, which calls stream.close(), which does some calls on loop requesting that it flush the buffer and then close the underlying socket. So far so good. ...but then immediately after this, the loop itself falls out of scope, and you lose. AFAICT from a quick skim of the asyncio code, the data will not be sent and the socket will not be closed (depending on kernel buffering etc.). (And even if you wrap everything with proper 'with' blocks, this is still true... asyncio event loops don't seem to have any API for "complete all work and then shut down"? Maybe I'm just missing it -- if not then this is possibly a rather serious bug in actual currently-existing asyncio. But for the present purposes, the point is that you really do need something like 'async with' around everything here to force the I/O to complete before handing things over to the gc -- you can't rely on the gc to do your I/O.) -n [1] https://github.com/kennethreitz/requests/issues/1390#issuecomment-225361421 -- Nathaniel J. Smith -- https://vorpus.org From dave at dabeaz.com Wed Jul 6 15:54:54 2016 From: dave at dabeaz.com (David Beazley) Date: Wed, 6 Jul 2016 14:54:54 -0500 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: Message-ID: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> > However, as far as I know curio doesn?t have the ability to schedule an operation in a synchronous manner by means of something like a Future. Is that correct? If there is no way in curio to spawn a task to occur later without having to await on it, then clearly there is no option but to allow coroutines in __aexit__ and finally: how else could curio operate? > Yes, Curio does not utilize Futures or callback functions for that matter. However, I think the real issue at hand might be much more subtle and weird. If I understand things correctly, the goal is to implement an asynchronous generator for feeding asynchronous iteration. This is a new kind of generator that runs inside of a coroutine (so, imagine a coroutine inside of another coroutine). Without seeing much more, I'd guess it would look something roughly akin to this: async def agen(): ... some sort of async generator ... async yield value # ???? Syntax ???? async def main(): async for x in agen(): ... There's some kind of underlying protocol driving the async iteration, but it's *different* than what is being used by the enclosing coroutines. Yes, there is a scheduler kernel (or event loop) that makes the outer coroutines run, but that scheduler is not driving the underlying iteration protocol of the async generator part. So, things get weird when stuff like this happens: async def main(): async for x in agen(): if x == STOP: break Here, the asynchronous generator makes it to a point, but never to completion because of the break statement. Instead, it gets garbage collected and all hell breaks loose back in the agen() function because what happens now? Especially if agen() uses finally or an async context manager: async def agen(): async with whatever: ... Assuming that this is getting to the heart of the issue, I spent some time pondering it this morning and almost wonder if it could be solved by "underthinking" the solution so to speak. For example, perhaps the __del__() method of an async-generator could just raise a RuntimeError if it's ever garbage collected before being properly terminated. Maybe you give asynchronous generators an async-close method to explicitly shut it down. So, you'd have to do this. async def main(): items = agen() async for x in items: if x == STOP: break await items.close() Maybe the async-close() method would merely raise AsyncGeneratorExit in the generator and not enforce any other kind of semantics other than having it continue to run as a coroutine as it did before (with the understanding that the purpose of the exception is to terminate eventually). Since forgetting that last close() step would be easy, naturally an async generator should support the asynchronous context-management protocol. async def main(): async with agen() as items: async for x in items: if x == STOP: break Perhaps the only thing missing at this point is a metaclass---or possibly a codec. I'm not sure. Yikes Cheers, Dave From yselivanov at gmail.com Wed Jul 6 16:12:58 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 6 Jul 2016 16:12:58 -0400 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> Message-ID: > On Jul 6, 2016, at 3:54 PM, David Beazley wrote: > > >> However, as far as I know curio doesn?t have the ability to schedule an operation in a synchronous manner by means of something like a Future. Is that correct? If there is no way in curio to spawn a task to occur later without having to await on it, then clearly there is no option but to allow coroutines in __aexit__ and finally: how else could curio operate? >> > > Yes, Curio does not utilize Futures or callback functions for that matter. However, I think the real issue at hand might be much more subtle and weird. > > If I understand things correctly, the goal is to implement an asynchronous generator for feeding asynchronous iteration. This is a new kind of generator that runs inside of a coroutine (so, imagine a coroutine inside of another coroutine). Without seeing much more, I'd guess it would look something roughly akin to this: > > async def agen(): > ... some sort of async generator ... > async yield value # ???? Syntax ???? I my current WIP branch I just use ?yield? inside ?async def?. That?s what I was going to propose to do in the PEP. > > async def main(): > async for x in agen(): > ... > > There's some kind of underlying protocol driving the async iteration, but it's *different* than what is being used by the enclosing coroutines. Yes, there is a scheduler kernel (or event loop) that makes the outer coroutines run, but that scheduler is not driving the underlying iteration protocol of the async generator part. So, things get weird when stuff like this happens: > > async def main(): > async for x in agen(): > if x == STOP: > break Good catch. [..] > Since forgetting that last close() step would be easy, naturally an async generator should support the asynchronous context-management protocol. > > async def main(): > async with agen() as items: > async for x in items: > if x == STOP: > break This is an interesting idea, but I wonder if instead of using ?async with? we can actually augment ?async for? to do the async finalization. We can add an __aiter_close__ special method which will return an awaitable. In this case, ?async for? can always look for that method and call it at the end of the iteration. Async generators will implement the method to make sure that ?finally? is always executed (any number of awaits in ?finally? is OK; ?yield? expressions cannot be used). Yury From Gerasimov-M-N at yandex.ru Wed Jul 6 18:36:27 2016 From: Gerasimov-M-N at yandex.ru (=?UTF-8?B?0JPQtdGA0LDRgdC40LzQvtCyINCc0LjRhdCw0LjQuw==?=) Date: Thu, 7 Jul 2016 01:36:27 +0300 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: Message-ID: <3ba41f2e-3d7f-6f55-4e49-9c1a9b46981a@yandex.ru> I tried to implement asyncronious generators based on asyncio recently, you can see result here: https://github.com/germn/aiogen I also faced problem with cleanup. First thing is where to call `.close` (or async `.aclose` for AG since it raises exception inside async function). While regular generator calls it inside `__del__` as I understand there's no guarantee AG's event loop wouldn't be closed at this moment. I think AG can be closed at the moment it's parent task done. Since this is non-async callback we can start task to call `.aclose`: https://github.com/germn/aiogen/blob/master/aiogen/agenerator.py#L35 Nothing awaits for cleanup task so we should sure that is would be finished before event loop is closed. I found nothings better than to decorate event loop's `.close` method: https://github.com/germn/aiogen/blob/master/aiogen/agenerator.py#L52 That all doesn't look like ideal solution, but it works. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jul 6 19:06:38 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jul 2016 16:06:38 -0700 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> Message-ID: On Wed, Jul 6, 2016 at 1:12 PM, Yury Selivanov wrote: > This is an interesting idea, but I wonder if instead of using ?async with? we can actually augment ?async for? to do the async finalization. > > We can add an __aiter_close__ special method which will return an awaitable. In this case, ?async for? can always look for that method and call it at the end of the iteration. Async generators will implement the method to make sure that ?finally? is always executed (any number of awaits in ?finally? is OK; ?yield? expressions cannot be used). I was wondering about that too. This is a fairly substantial change to how iterators work, though -- currently, it's totally legal and sometimes very useful to do things like it = open("...") # Discard header line (first non-commented line) for line in it: if not line.startswith("#"): break for body_line in it: ... or nested for loops like: for section_header_line in it: section_info = parse_section_header_line(section_header_line) if section_info.type == "multi-line": for body_line in it: if body_line == "-- end of section --": break else: ... I guess there are a few different options in this design space -- one could have a dedicated for+with syntax ("async forthwith"?), or an opt-out utility like for section_header_line in it: for body_line in protect_from_close(it): ... where protect_from_close returns a wrapper object that intercepts __iter_close__ and ignores it, while passing through other method calls (__next__/send/throw). -n -- Nathaniel J. Smith -- https://vorpus.org From yselivanov at gmail.com Wed Jul 6 20:25:38 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 6 Jul 2016 20:25:38 -0400 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> Message-ID: > On Jul 6, 2016, at 7:06 PM, Nathaniel Smith wrote: > > On Wed, Jul 6, 2016 at 1:12 PM, Yury Selivanov wrote: >> This is an interesting idea, but I wonder if instead of using ?async with? we can actually augment ?async for? to do the async finalization. >> >> We can add an __aiter_close__ special method which will return an awaitable. In this case, ?async for? can always look for that method and call it at the end of the iteration. Async generators will implement the method to make sure that ?finally? is always executed (any number of awaits in ?finally? is OK; ?yield? expressions cannot be used). > > I was wondering about that too. This is a fairly substantial change to > how iterators work, though -- currently, it's totally legal and > sometimes very useful to do things like > > it = open("...") > # Discard header line (first non-commented line) > for line in it: > if not line.startswith("#"): > break > for body_line in it: > ? Right. __aiter_close__ won?t work. The problem is that the GC can?t execute async code, and we don?t have any control over GC. What if we add a mechanism to control how async generators (AG) are destructed. Let?s say we add new function to the sys module - `sys.set_async_generator_finalizer(finalizer)`. We already have sys.set_coroutine_wrapper(), so this isn?t something unprecedented. With this new function, when an AG is about to be finalized, the interpreter will resurrect it and call the `finalizer`. The finalizer function will be installed by an event loop, and will execute ?await AG.aclose()? in the context of that loop. We can issue a ResourceWarning if an AG (not fully exhausted) is GCed and there is no async finalizer. Yury From glyph at twistedmatrix.com Wed Jul 6 20:47:10 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 6 Jul 2016 17:47:10 -0700 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> Message-ID: <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> > On Jul 6, 2016, at 5:25 PM, Yury Selivanov wrote: > > The problem is that the GC can?t execute async code, and we don?t have any control over GC. What if we add a mechanism to control how async generators (AG) are destructed. Let?s say we add new function to the sys module - `sys.set_async_generator_finalizer(finalizer)`. We already have sys.set_coroutine_wrapper(), so this isn?t something unprecedented. There isn't just one event loop though, and what trampoline to attach a dying coroutine to depends heavily on what event loop it came from. It seems like a single global value for this in 'sys' would just be ... wrong. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Wed Jul 6 21:08:31 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 6 Jul 2016 21:08:31 -0400 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> Message-ID: <8D989DA8-9109-44D4-BB03-936FF6A10B61@gmail.com> > On Jul 6, 2016, at 8:47 PM, Glyph Lefkowitz wrote: > > >> On Jul 6, 2016, at 5:25 PM, Yury Selivanov wrote: >> >> The problem is that the GC can?t execute async code, and we don?t have any control over GC. What if we add a mechanism to control how async generators (AG) are destructed. Let?s say we add new function to the sys module - `sys.set_async_generator_finalizer(finalizer)`. We already have sys.set_coroutine_wrapper(), so this isn?t something unprecedented. > > There isn't just one event loop though, and what trampoline to attach a dying coroutine to depends heavily on what event loop it came from. It seems like a single global value for this in 'sys' would just be ... wrong. But there can only be one currently running event loop per thread... Another way is to add sys.set_async_generator_wrapper() (and a getter, so that loops can maintain a stack of them). With it a running event loop can create a weak ref to a generator that is created in a coroutine that the loop is currently running. And with a weakref it can later finalize the generator. Yury -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jul 6 21:10:04 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jul 2016 18:10:04 -0700 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> Message-ID: On Wed, Jul 6, 2016 at 5:47 PM, Glyph Lefkowitz wrote: > > On Jul 6, 2016, at 5:25 PM, Yury Selivanov wrote: > > The problem is that the GC can?t execute async code, and we don?t have any > control over GC. What if we add a mechanism to control how async generators > (AG) are destructed. Let?s say we add new function to the sys module - > `sys.set_async_generator_finalizer(finalizer)`. We already have > sys.set_coroutine_wrapper(), so this isn?t something unprecedented. > > > There isn't just one event loop though, and what trampoline to attach a > dying coroutine to depends heavily on what event loop it came from. It > seems like a single global value for this in 'sys' would just be ... wrong. I guess one could standardize a "get finalizer" protocol for coroutine runners, where async generators (or other objects with the same problem, I guess) would do the equivalent of class AsyncGenerator: async def __anext__(self, *args, **kwargs): if not hasattr(self, "_finalizer"): self._finalizer = (yield GET_FINALIZER_SENTINEL) ... def __del__(self): self._finalizer.run_until_complete(self.aclose()) i.e., we provide some standard mechanism for coroutine code to take a reference on the coroutine runner when starting up (when they do have access to it), and then re-enter it for cleanup. This feels a bit elaborate, though, and produces some pretty convoluted control flow. ...does it actually work to re-enter a main loop from inside a __del__ callback? It seems like you could get into really nasty states with multiple nested __del__ calls, or if a single sweep detects multiple pieces of garbage with __del__ methods, then some of those __del__ calls could be delayed indefinitely while the first __del__ runs. Is the cycle collector even re-entrant? -n -- Nathaniel J. Smith -- https://vorpus.org From yselivanov at gmail.com Wed Jul 6 21:17:48 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 6 Jul 2016 21:17:48 -0400 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> Message-ID: > ...does it actually work to re-enter a main loop from inside a __del__ > callback? It seems like you could get into really nasty states with > multiple nested __del__ calls, or if a single sweep detects multiple > pieces of garbage with __del__ methods, then some of those __del__ > calls could be delayed indefinitely while the first __del__ runs. Is > the cycle collector even re-entrant? We can have a flag on the async gen object to make sure that we run the finalizer only once. The finalizer will likely schedule async_gen.aclose() coroutine which will ensure a strong ref to the gen until it is closed. This can actually work.. ;) Yury From njs at pobox.com Wed Jul 6 21:44:21 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 6 Jul 2016 18:44:21 -0700 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> Message-ID: On Wed, Jul 6, 2016 at 6:17 PM, Yury Selivanov wrote: > >> ...does it actually work to re-enter a main loop from inside a __del__ >> callback? It seems like you could get into really nasty states with >> multiple nested __del__ calls, or if a single sweep detects multiple >> pieces of garbage with __del__ methods, then some of those __del__ >> calls could be delayed indefinitely while the first __del__ runs. Is >> the cycle collector even re-entrant? > > We can have a flag on the async gen object to make sure that we run the finalizer only once. The finalizer will likely schedule async_gen.aclose() coroutine which will ensure a strong ref to the gen until it is closed. This can actually work.. ;) Hmm, if the strategy is to schedule the work to happen outside of the actual __del__ call, then I think this is back to assuming that all coroutine runners are immortal and always running. Is that an assumption you're comfortable with? -n -- Nathaniel J. Smith -- https://vorpus.org From yselivanov at gmail.com Wed Jul 6 22:29:53 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 6 Jul 2016 22:29:53 -0400 Subject: [Async-sig] Asynchronous cleanup is a problem In-Reply-To: References: <243B67B1-31C9-448D-BD84-4EBAD2584C27@dabeaz.com> <2FAAF33F-E56A-4A56-B406-71AE684D1BBA@twistedmatrix.com> Message-ID: <5011CB73-8F7C-4E6E-844C-B45A445BA57A@gmail.com> > On Jul 6, 2016, at 9:44 PM, Nathaniel Smith wrote: > > On Wed, Jul 6, 2016 at 6:17 PM, Yury Selivanov wrote: >> >>> ...does it actually work to re-enter a main loop from inside a __del__ >>> callback? It seems like you could get into really nasty states with >>> multiple nested __del__ calls, or if a single sweep detects multiple >>> pieces of garbage with __del__ methods, then some of those __del__ >>> calls could be delayed indefinitely while the first __del__ runs. Is >>> the cycle collector even re-entrant? >> >> We can have a flag on the async gen object to make sure that we run the finalizer only once. The finalizer will likely schedule async_gen.aclose() coroutine which will ensure a strong ref to the gen until it is closed. This can actually work.. ;) > > Hmm, if the strategy is to schedule the work to happen outside of the > actual __del__ call, then I think this is back to assuming that all > coroutine runners are immortal and always running. Is that an > assumption you're comfortable with? No need to require coroutine runners to be immortal. If a finalizer finds out that its event loop is closed, it does nothing. The interpreter will then issue a ResourceWarning if the generator wasn?t properly closed. This is how a finalizer for asyncio event loop might look like: def _finalize_gen(self, gen): if not self.is_closed(): self.create_task(gen.aclose()) And this is how asyncio loop might set it up: class AsyncioEventLoop: def run_forever(): ... old_finalizer = sys.get_async_generator_finalizer() sys.set_async_generator_finalizer(self._finalize_gen) try: while True: self._run_once() ? finally: ? sys.set_async_generator_finalizer(old_finalizer) Why do I think that it?s OK that some async generators might end up not being properly closed? Because this can already happen to coroutines: async def coro1(): try: print('try') await asyncio.sleep(1) finally: await asyncio.sleep(0) print('finally') async def coro2(): await asyncio.sleep(0) loop = asyncio.get_event_loop() loop.create_task(coro1()) loop.run_until_complete(coro2()) In other words, an event loop might stop or be interrupted, and there is no way to guarantee that all coroutines will be finalized properly in that case. To address Glyph?s point about many event loops in one process: a finalizer set with set_async_generator_finalizer() (which should be thread specific, same as set_coroutine_wrapper) can actually be assigned to the async generator when it?s instantiated. Yury From yselivanov at gmail.com Fri Jul 29 12:18:10 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 29 Jul 2016 12:18:10 -0400 Subject: [Async-sig] PEP: asynchronous generators Message-ID: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Hi, I have been working on a PEP to add asynchronous generators to Python. The PEP is now ready for a review. It would be great to hear some initial feedback from async-sig, before I post it to python-ideas. I have a complete and working reference implementation of everything that PEP proposes here: https://github.com/1st1/cpython/tree/async_gen PEP: XXX Title: Asynchronous Generators Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov Discussions-To: Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Jul-2016 Python-Version: 3.6 Post-History: Abstract ======== PEP 492 introduced support for native coroutines and ``async``/``await`` syntax to Python 3.5. It is proposed here to extend Python's asynchronous capabilities by adding support for *asynchronous generators*. Rationale and Goals =================== Regular generators (introduced in PEP 255) enabled an elegant way of writing complex *data producers* and have them behave like an iterator. However, currently there is no equivalent concept for the *asynchronous iteration protocol* (``async for``). This makes writing asynchronous data producers unnecessarily complex, as one must define a class that implements ``__aiter__`` to be able to use it in an ``async for`` statement. Essentially, the goals and rationale for PEP 255, applied to the asynchronous execution case, hold true for this proposal as well. Performance is an additional point for this proposal: in our testing of the reference implementation, asynchronous generators are *2x* faster than an equivalent implemented as an asynchronous iterator. As an illustration of the code quality improvement, consider the following class that prints numbers with a given delay once iterated:: class ticker: """Print numbers from 0 to `to` every `delay` seconds.""" def __init__(self, delay, to): self.delay = delay self.i = 0 self.to = to def __aiter__(self): return self async def __anext__(self): i = self.i if i >= self.to: raise StopAsyncIteration self.i += 1 if i: await asyncio.sleep(self.delay) return i The same can be implemented as a much simpler asynchronous generator:: async def ticker(delay, to): """Print numbers from 0 to `to` every `delay` seconds.""" i = 0 while i < to: yield i i += 1 await asyncio.sleep(delay) Specification ============= This proposal introduces the concept of *asynchronous generators* to Python. This specification presumes knowledge of the implementation of generators and coroutines in Python (PEP 342, PEP 380 and PEP 492). Asynchronous Generators ----------------------- A Python *generator* is any function containing one or more ``yield`` expressions:: def func(): # a function return def genfunc(): # a generator function yield We propose to use the same approach to define *asynchronous generators*:: async def coro(): # a coroutine function await smth() async def asyncgen(): # an asynchronous generator function await smth() yield val The result of calling an *asynchronous generator function* is an *asynchronous generator object*, which implements the asynchronous iteration protocol defined in PEP 492. It is a ``SyntaxError`` to have a non-empty ``return`` statement in an asynchronous generator. Support for Asynchronous Iteration Protocol ------------------------------------------- The protocol requires two special methods to be implemented: 1. An ``__aiter__`` method returning an *asynchronous iterator*. 2. An ``__anext__`` method returning an *awaitable* object, which uses ``StopIteration`` exception to "yield" values, and ``StopAsyncIteration`` exception to signal the end of the iteration. Asynchronous generators define both of these methods:: async def genfunc(): yield 1 yield 2 gen = genfunc() assert gen.__aiter__() is gen assert await gen.__anext__() == 1 assert await gen.__anext__() == 2 with assertRaises(StopAsyncIteration): await gen.__anext__() Finalization ------------ PEP 492 requires an event loop or a scheduler to run coroutines. Because asynchronous generators are meant to be used from coroutines, they also require an event loop to run and finalize them. Asynchronous generators can have ``try..finally`` blocks, as well as ``async with``. It is important to provide a guarantee that, even when partially iterated, and then garbage collected, generators can be safely finalized. For example:: async def square_series(con, to): async with con.transaction(): cursor = con.cursor( 'SELECT generate_series(0, $1) AS i', to) async for row in cursor: yield row['i'] ** 2 async for i in square_series(con, 100): if i == 100: break The above code defines an asynchronous generator that uses ``async with`` to iterate over a database cursor in a transaction. The generator is then iterated over with ``async for``, which interrupts the iteration at some point. The ``square_series()`` generator will then be garbage collected, and without a mechanism to asynchronously close the generator, Python interpreter would not be able to do anything. To solve this problem we propose to do the following: 1. Implement an ``aclose`` method on asynchronous generators returning a special *awaitable*. When awaited it throws a ``GeneratorExit`` into the suspended generator and iterates over it until either a ``GeneratorExit`` or a ``StopAsyncIteration`` occur. This is very similar to what the ``close()`` method does to regular Python generators, except that an event loop is required to execute ``aclose()``. 2. Raise a ``RuntimeError``, when an asynchronous generator executes a ``yield`` expression in its ``finally`` block (using ``await`` is fine, though):: async def gen(): try: yield finally: yield # Cannot use 'yield' await asyncio.sleep(1) # Can use 'await' 3. Add two new methods to the ``sys`` module: ``set_asyncgen_finalizer`` and ``get_asyncgen_finalizer``. The idea behind ``sys.set_asyncgen_finalizer`` is to allow event loops to handle generators finalization, so that the end user does not need to care about the finalization problem, and it just works. When an asynchronous generator is iterated for the first time, it stores a reference to the current finalizer. If there is none, a ``RuntimeError`` is raised. This provides a strong guarantee that every asynchronous generator object will always have a finalizer installed by the correct event loop. When an asynchronous generator is about to be garbage collected, it calls its cached finalizer. The assumption is that the finalizer will schedule an ``aclose()`` call with the loop that was active when the iteration started. For instance, here is how asyncio can be modified to allow safe finalization of asynchronous generators:: # asyncio/base_events.py class BaseEventLoop: def run_forever(self): ... old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(self._finalize_asyncgen) try: ... finally: sys.set_asyncgen_finalizer(old_finalizer) ... def _finalize_asyncgen(self, gen): self.create_task(gen.aclose()) ``sys.set_asyncgen_finalizer`` is thread-specific, so several event loops running in parallel threads can use it safely. Asynchronous Generator Object ----------------------------- The object is modeled after the standard Python generator object. Essentially, the behaviour of asynchronous generators is designed to replicate the behaviour of synchronous generators, with the only difference in that the API is asynchronous. The following methods and properties are defined: 1. ``agen.__aiter__()``: Returns ``agen``. 2. ``agen.__anext__()``: Returns an *awaitable*, that performs one asynchronous generator iteration when awaited. 3. ``agen.anext(val)``: Returns an *awaitable*, that pushes the ``val`` object in the ``agen`` generator. When the ``agen`` has not yet been iterated, ``val`` must be ``None``. Example:: async def gen(): await asyncio.sleep(0.1) v = yield 42 print(v) await asyncio.sleep(0.1) g = gen() await g.send(None) # Will return 42 await g.send('hello') # Will print 'hello' and # raise StopAsyncIteration # (after sleeping for 0.1 seconds) 4. ``agen.athrow(typ, [val, [tb]])``: Returns an *awaitable*, that throws an exception into the ``agen`` generator. Example:: async def gen(): try: await asyncio.sleep(0.1) yield 'hello' except ZeroDivisionError: await asyncio.sleep(0.2) yield 'world' g = gen() v = await g.asend(None) print(v) # Will print 'hello' after sleeping 0.1s v = await g.athrow(ZeroDivisionError) print(v) # Will print 'world' after sleeping 0.2s 5. ``agen.aclose()``: Returns an *awaitable*, that throws a ``GeneratorExit`` exception into the generator. The *awaitable* can either return a yielded value, if ``agen`` handled the exception, or ``agen`` will be closed and the exception will propagate back to the caller. 6. ``agen.__name__`` and ``agen.__qualname__``: readable and writable name and qualified name attributes. 7. ``agen.ag_await``: The object that ``agen`` is currently awaiting on, or ``None``. 8. ``agen.ag_frame``, ``agen.ag_running``, and ``agen.ag_code``: defined in the same way as similar attributes of standard generators. New Standard Library Functions and Types ---------------------------------------- 1. ``types.AsyncGeneratorType`` -- type of asynchronous generator object. 2. ``sys.set_asyncgen_finalizer()`` and ``sys.get_asyncgen_finalizer()`` methods to set up asynchronous generators finalizers in event loops. 3. ``inspect.isasyncgen()`` and ``inspect.isasyncgenfunction()`` introspection functions. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. In Python 3.5 it is a ``SyntaxError`` to define an ``async def`` function with a ``yield`` expression inside, therefore it's safe to introduce asynchronous generators in 3.6. Performance =========== Regular Generators ------------------ There is no performance degradation for regular generators. The following micro benchmark runs at the same speed on CPython with and without asynchronous generators:: def gen(): i = 0 while i < 100000000: yield i i += 1 list(gen()) Improvements over asynchronous iterators ---------------------------------------- The following micro-benchmark shows that asynchronous generators are about **2x faster** than asynchronous iterators implemented in pure Python: async def agen(): i = 0 while i < N: yield i i += 1 class AIter: def __init__(self): self.i = 0 def __aiter__(self): return self async def __anext__(self): i = self.i if i >= N: raise StopAsyncIteration self.i += 1 return i Design Considerations ===================== ``aiter()`` and ``anext()`` builtins ------------------------------------ Originally PEP 492 defined ``__aiter__`` as a method that should return an *awaitable* object, resulting in an asynchronous iterator. However, in CPython 3.5.2, ``__aiter__`` was redefined to return asynchronous iterators directly. To avoid breaking backwards compatibility, it was decided that Python 3.6 will support both ways: ``__aiter__`` can still return an *awaitable* with a ``DeprecationWarning`` being issued. Because of this dual nature of ``__aiter__`` in Python 3.6, we cannot add a synchronous implementation of ``aiter()`` built-in. Therefore, it is proposed to wait until Python 3.7. Asynchronous list/dict/set comprehensions ----------------------------------------- Syntax for asynchronous comprehensions is unrelated to the asynchronous generators machinery, and should be considered in a separate PEP. Asynchronous ``yield from`` --------------------------- While it is theoretically possible to implement ``yield from`` support for asynchronous generators, it would require a serious redesign of the generator implementation. ``yield from`` is also less critical for asynchronous generators, since there is no need provide a mechanism of implementing another coroutines protocol on top of coroutines. To compose asynchronous generators a simple ``async for`` loop can be used:: async def g1(): yield 1 yield 2 async def g2(): async for v in g1(): yield v Why the ``asend`` and ``athrow`` methods are necessary ------------------------------------------------------ They make it possible to implement concepts similar to ``contextlib.contextmanager`` using asynchronous generators. For instance, with the proposed design, it is possible to implement the following pattern:: @async_context_manager async def ctx(): await open() try: yield finally: await close() async with ctx(): await ... Another reason is that it is possible to push data and throw exceptions into asynchronous generators using the object returned from ``__anext__`` object, but it is hard to do that correctly. Adding explicit ``asend`` and ``athrow`` will pave a safe way to accomplish that. Example ======= A working example with the current reference implementation (will print numbers from 0 to 9 with one second delay):: async def ticker(delay, to): i = 0 while i < to: yield i i += 1 await asyncio.sleep(delay) async def run(): async for i in ticker(1, 10): print(i) import asyncio loop = asyncio.get_event_loop() try: loop.run_until_complete(run()) finally: loop.close() Implementation ============== The complete reference implementation is available at [1]_. References ========== .. [1] https://github.com/1st1/cpython/tree/async_gen Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: Thank you! Yury From brett at python.org Fri Jul 29 13:25:52 2016 From: brett at python.org (Brett Cannon) Date: Fri, 29 Jul 2016 17:25:52 +0000 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: On Fri, 29 Jul 2016 at 09:18 Yury Selivanov wrote: > Hi, > > I have been working on a PEP to add asynchronous generators to > Python. The PEP is now ready for a review. Woohoo! > It would be great to > hear some initial feedback from async-sig, before I post it to > python-ideas. > > I have a complete and working reference implementation of > everything that PEP proposes here: > > https://github.com/1st1/cpython/tree/async_gen > > > PEP: XXX > Title: Asynchronous Generators > Version: $Revision$ > Last-Modified: $Date$ > Author: Yury Selivanov > Discussions-To: > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 28-Jul-2016 > Python-Version: 3.6 > Post-History: > > > Abstract > ======== > > PEP 492 introduced support for native coroutines and ``async``/``await`` > syntax to Python 3.5. It is proposed here to extend Python's > asynchronous capabilities by adding support for > *asynchronous generators*. > > > Rationale and Goals > =================== > > Regular generators (introduced in PEP 255) enabled an elegant way of > writing complex *data producers* and have them behave like an iterator. > > However, currently there is no equivalent concept for the *asynchronous > iteration protocol* (``async for``). This makes writing asynchronous > data producers unnecessarily complex, as one must define a class that > implements ``__aiter__`` to be able to use it in an ``async for`` > statement. > > Essentially, the goals and rationale for PEP 255, applied to the > asynchronous execution case, hold true for this proposal as well. > > Performance is an additional point for this proposal: in our testing of > the reference implementation, asynchronous generators are *2x* faster > than an equivalent implemented as an asynchronous iterator. > Another motivation is that types.coroutine becomes purely a backwards-compatibility/low-level thing that is no longer required for event loop frameworks to use on their generators which provide their async API. So now an async framework can be written entirely in terms of def and async def and not def and @types.coroutine. > > As an illustration of the code quality improvement, consider the > following class that prints numbers with a given delay once iterated:: > > class ticker: > """Print numbers from 0 to `to` every `delay` seconds.""" > > def __init__(self, delay, to): > self.delay = delay > self.i = 0 > self.to = to > > def __aiter__(self): > return self > > async def __anext__(self): > i = self.i > if i >= self.to: > raise StopAsyncIteration > self.i += 1 > if i: > await asyncio.sleep(self.delay) > return i > > > The same can be implemented as a much simpler asynchronous generator:: > > async def ticker(delay, to): > """Print numbers from 0 to `to` every `delay` seconds.""" > i = 0 > while i < to: > yield i > i += 1 > await asyncio.sleep(delay) > > > Specification > ============= > > This proposal introduces the concept of *asynchronous generators* to > Python. > > This specification presumes knowledge of the implementation of > generators and coroutines in Python (PEP 342, PEP 380 and PEP 492). > > > Asynchronous Generators > ----------------------- > > A Python *generator* is any function containing one or more ``yield`` > expressions:: > > def func(): # a function > return > > def genfunc(): # a generator function > yield > > We propose to use the same approach to define > *asynchronous generators*:: > > async def coro(): # a coroutine function > await smth() > > async def asyncgen(): # an asynchronous generator function > await smth() > yield val > > The result of calling an *asynchronous generator function* is > an *asynchronous generator object*, which implements the asynchronous > iteration protocol defined in PEP 492. > > It is a ``SyntaxError`` to have a non-empty ``return`` statement in an > asynchronous generator. > > > Support for Asynchronous Iteration Protocol > ------------------------------------------- > > The protocol requires two special methods to be implemented: > > 1. An ``__aiter__`` method returning an *asynchronous iterator*. > 2. An ``__anext__`` method returning an *awaitable* object, which uses > ``StopIteration`` exception to "yield" values, and > ``StopAsyncIteration`` exception to signal the end of the iteration. > I assume this is in place of __iter__() and __next__(), i.e. an async generator won't have both defined? > > Asynchronous generators define both of these methods:: > > async def genfunc(): > yield 1 > yield 2 > > gen = genfunc() > > assert gen.__aiter__() is gen > > assert await gen.__anext__() == 1 > assert await gen.__anext__() == 2 > > with assertRaises(StopAsyncIteration): > await gen.__anext__() > > > Finalization > ------------ > > PEP 492 requires an event loop or a scheduler to run coroutines. > Because asynchronous generators are meant to be used from coroutines, > they also require an event loop to run and finalize them. > > Asynchronous generators can have ``try..finally`` blocks, as well as > ``async with``. It is important to provide a guarantee that, even > when partially iterated, and then garbage collected, generators can > be safely finalized. For example:: > > async def square_series(con, to): > async with con.transaction(): > cursor = con.cursor( > 'SELECT generate_series(0, $1) AS i', to) > async for row in cursor: > yield row['i'] ** 2 > > async for i in square_series(con, 100): > if i == 100: > break > > The above code defines an asynchronous generator that uses > ``async with`` to iterate over a database cursor in a transaction. > The generator is then iterated over with ``async for``, which interrupts > the iteration at some point. > > The ``square_series()`` generator will then be garbage collected, > and without a mechanism to asynchronously close the generator, Python > interpreter would not be able to do anything. > > To solve this problem we propose to do the following: > > 1. Implement an ``aclose`` method on asynchronous generators > returning a special *awaitable*. When awaited it > throws a ``GeneratorExit`` into the suspended generator and > iterates over it until either a ``GeneratorExit`` or > a ``StopAsyncIteration`` occur. > > This is very similar to what the ``close()`` method does to regular > Python generators, except that an event loop is required to execute > ``aclose()``. > I'm going to ask this now instead of later when there's more motivation behind this question: do we need to append "a" to every async method we have? If asynchronous generators won't have a normal close() then why can't it just be close(), especially if people are not going to be calling it directly and instead it will be event loops? I'm just leery of codifying this practice of prepending "a" to every async method or function and ending up in a future where I get tired of a specific letter of the alphabet. > > 2. Raise a ``RuntimeError``, when an asynchronous generator executes > a ``yield`` expression in its ``finally`` block (using ``await`` > is fine, though):: > > async def gen(): > try: > yield > finally: > yield # Cannot use 'yield' > await asyncio.sleep(1) # Can use 'await' > > 3. Add two new methods to the ``sys`` module: > ``set_asyncgen_finalizer`` and ``get_asyncgen_finalizer``. > > The idea behind ``sys.set_asyncgen_finalizer`` is to allow event > loops to handle generators finalization, so that the end user > does not need to care about the finalization problem, and it just > works. > When an asynchronous generator is iterated for the first time, > it stores a reference to the current finalizer. If there is none, > a ``RuntimeError`` is raised. This provides a strong guarantee that > every asynchronous generator object will always have a finalizer > installed by the correct event loop. > When an asynchronous generator is about to be garbage collected, > it calls its cached finalizer. The assumption is that the finalizer > will schedule an ``aclose()`` call with the loop that was active > when the iteration started. > > For instance, here is how asyncio can be modified to allow > safe finalization of asynchronous generators:: > > # asyncio/base_events.py > > class BaseEventLoop: > > def run_forever(self): > ... > old_finalizer = sys.get_asyncgen_finalizer() > sys.set_asyncgen_finalizer(self._finalize_asyncgen) > try: > ... > finally: > sys.set_asyncgen_finalizer(old_finalizer) > ... > > def _finalize_asyncgen(self, gen): > self.create_task(gen.aclose()) > > ``sys.set_asyncgen_finalizer`` is thread-specific, so several event > loops running in parallel threads can use it safely. > > > Asynchronous Generator Object > ----------------------------- > > The object is modeled after the standard Python generator object. > Essentially, the behaviour of asynchronous generators is designed > to replicate the behaviour of synchronous generators, with the only > difference in that the API is asynchronous. > > The following methods and properties are defined: > > 1. ``agen.__aiter__()``: Returns ``agen``. > > 2. ``agen.__anext__()``: Returns an *awaitable*, that performs one > asynchronous generator iteration when awaited. > > 3. ``agen.anext(val)``: Returns an *awaitable*, that pushes the > ``val`` object in the ``agen`` generator. When the ``agen`` has > not yet been iterated, ``val`` must be ``None``. > How is this different from an async send()? I see an asend() used in the example below but not anext(). > > Example:: > > async def gen(): > await asyncio.sleep(0.1) > v = yield 42 > print(v) > await asyncio.sleep(0.1) > > g = gen() > await g.send(None) # Will return 42 > await g.send('hello') # Will print 'hello' and > # raise StopAsyncIteration > # (after sleeping for 0.1 seconds) > > 4. ``agen.athrow(typ, [val, [tb]])``: Returns an *awaitable*, that > throws an exception into the ``agen`` generator. > > Example:: > > async def gen(): > try: > await asyncio.sleep(0.1) > yield 'hello' > except ZeroDivisionError: > await asyncio.sleep(0.2) > yield 'world' > > g = gen() > v = await g.asend(None) > print(v) # Will print 'hello' after sleeping 0.1s > v = await g.athrow(ZeroDivisionError) > print(v) # Will print 'world' after sleeping 0.2s > > 5. ``agen.aclose()``: Returns an *awaitable*, that throws a > ``GeneratorExit`` exception into the generator. The *awaitable* can > either return a yielded value, if ``agen`` handled the exception, > or ``agen`` will be closed and the exception will propagate back > to the caller. > > 6. ``agen.__name__`` and ``agen.__qualname__``: readable and writable > name and qualified name attributes. > > 7. ``agen.ag_await``: The object that ``agen`` is currently awaiting on, > or ``None``. > That's an interesting addition. I like it! There's no equivalent on normal generators, correct? > > 8. ``agen.ag_frame``, ``agen.ag_running``, and ``agen.ag_code``: > defined in the same way as similar attributes of standard generators. > > > New Standard Library Functions and Types > ---------------------------------------- > > 1. ``types.AsyncGeneratorType`` -- type of asynchronous generator > object. > > 2. ``sys.set_asyncgen_finalizer()`` and ``sys.get_asyncgen_finalizer()`` > methods to set up asynchronous generators finalizers in event loops. > > 3. ``inspect.isasyncgen()`` and ``inspect.isasyncgenfunction()`` > introspection functions. > > > Backwards Compatibility > ----------------------- > > The proposal is fully backwards compatible. > > In Python 3.5 it is a ``SyntaxError`` to define an ``async def`` > function with a ``yield`` expression inside, therefore it's safe to > introduce asynchronous generators in 3.6. > > > Performance > =========== > > Regular Generators > ------------------ > > There is no performance degradation for regular generators. > The following micro benchmark runs at the same speed on CPython with > and without asynchronous generators:: > > def gen(): > i = 0 > while i < 100000000: > yield i > i += 1 > > list(gen()) > > > Improvements over asynchronous iterators > ---------------------------------------- > > The following micro-benchmark shows that asynchronous generators > are about **2x faster** than asynchronous iterators implemented in > pure Python: > > async def agen(): > i = 0 > while i < N: > yield i > i += 1 > > > class AIter: > def __init__(self): > self.i = 0 > > def __aiter__(self): > return self > > async def __anext__(self): > i = self.i > if i >= N: > raise StopAsyncIteration > self.i += 1 > return i > > > Design Considerations > ===================== > > > ``aiter()`` and ``anext()`` builtins > ------------------------------------ > > Originally PEP 492 defined ``__aiter__`` as a method that should > return an *awaitable* object, resulting in an asynchronous iterator. > > However, in CPython 3.5.2, ``__aiter__`` was redefined to return > asynchronous iterators directly. To avoid breaking backwards > compatibility, it was decided that Python 3.6 will support both > ways: ``__aiter__`` can still return an *awaitable* with > a ``DeprecationWarning`` being issued. > > Because of this dual nature of ``__aiter__`` in Python 3.6, we cannot > add a synchronous implementation of ``aiter()`` built-in. Therefore, > it is proposed to wait until Python 3.7. > > > Asynchronous list/dict/set comprehensions > ----------------------------------------- > > Syntax for asynchronous comprehensions is unrelated to the asynchronous > generators machinery, and should be considered in a separate PEP. > > > Asynchronous ``yield from`` > --------------------------- > > While it is theoretically possible to implement ``yield from`` support > for asynchronous generators, it would require a serious redesign of the > generator implementation. > > ``yield from`` is also less critical for asynchronous generators, since > there is no need provide a mechanism of implementing another coroutines > protocol on top of coroutines. To compose asynchronous generators a > simple ``async for`` loop can be used:: > > async def g1(): > yield 1 > yield 2 > > async def g2(): > async for v in g1(): > yield v > > > Why the ``asend`` and ``athrow`` methods are necessary > ------------------------------------------------------ > > They make it possible to implement concepts similar to > ``contextlib.contextmanager`` using asynchronous generators. > For instance, with the proposed design, it is possible to implement > the following pattern:: > > @async_context_manager > async def ctx(): > await open() > try: > yield > finally: > await close() > > async with ctx(): > await ... > > Another reason is that it is possible to push data and throw exceptions > into asynchronous generators using the object returned from ``__anext__`` > object, but it is hard to do that correctly. Adding explicit ``asend`` > and ``athrow`` will pave a safe way to accomplish that. > > > Example > ======= > > A working example with the current reference implementation (will > print numbers from 0 to 9 with one second delay):: > > async def ticker(delay, to): > i = 0 > while i < to: > yield i > i += 1 > await asyncio.sleep(delay) > > > async def run(): > async for i in ticker(1, 10): > print(i) > > > import asyncio > loop = asyncio.get_event_loop() > try: > loop.run_until_complete(run()) > finally: > loop.close() > > > Implementation > ============== > > The complete reference implementation is available at [1]_. > > > References > ========== > > .. [1] https://github.com/1st1/cpython/tree/async_gen > > > Copyright > ========= > > This document has been placed in the public domain. > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > > Thank you! > Yury > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Fri Jul 29 13:46:14 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 29 Jul 2016 13:46:14 -0400 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: Thanks a lot for the feedback, Brett! Comments inlined below: > On Jul 29, 2016, at 1:25 PM, Brett Cannon wrote: > [..] > > Performance is an additional point for this proposal: in our testing of > the reference implementation, asynchronous generators are *2x* faster > than an equivalent implemented as an asynchronous iterator. > > Another motivation is that types.coroutine becomes purely a backwards-compatibility/low-level thing that is no longer required for event loop frameworks to use on their generators which provide their async API. So now an async framework can be written entirely in terms of def and async def and not def and @types.coroutine. A slight misunderstanding here: @types.coroutine turns a normal old-style generator into an awaitable object. That isn?t directly related to asynchronous iteration. Frameworks like curio will continue using @types.coroutine to implement future-like objects in event loops. [..] > > Support for Asynchronous Iteration Protocol > ------------------------------------------- > > The protocol requires two special methods to be implemented: > > 1. An ``__aiter__`` method returning an *asynchronous iterator*. > 2. An ``__anext__`` method returning an *awaitable* object, which uses > ``StopIteration`` exception to "yield" values, and > ``StopAsyncIteration`` exception to signal the end of the iteration. > > I assume this is in place of __iter__() and __next__(), i.e. an async generator won't have both defined? Correct, async generators don?t have __iter__ and __next__. They can only be iterated with an `async for`. [..] > > To solve this problem we propose to do the following: > > 1. Implement an ``aclose`` method on asynchronous generators > returning a special *awaitable*. When awaited it > throws a ``GeneratorExit`` into the suspended generator and > iterates over it until either a ``GeneratorExit`` or > a ``StopAsyncIteration`` occur. > > This is very similar to what the ``close()`` method does to regular > Python generators, except that an event loop is required to execute > ``aclose()``. > > I'm going to ask this now instead of later when there's more motivation behind this question: do we need to append "a" to every async method we have? If asynchronous generators won't have a normal close() then why can't it just be close(), especially if people are not going to be calling it directly and instead it will be event loops? I'm just leery of codifying this practice of prepending "a" to every async method or function and ending up in a future where I get tired of a specific letter of the alphabet. I decided to use the prefix because we already use it in magic method names: __anext__ and __aiter__. I think it also makes it easier to understand the API of async generators (and understand how it?s different from sync generators API). And while it?s entirely possible to drop the ?a? for async generator API, it?s not so simple for other cases. Later, for Python 3.7, we might consider adding ?aiter()? and ?anext()? builtins, for which we?d have to use ?a? or ?async? prefix (we can?t reuse 'iter()' and 'next()? for async generators). [..] > > 3. ``agen.anext(val)``: Returns an *awaitable*, that pushes the > ``val`` object in the ``agen`` generator. When the ``agen`` has > not yet been iterated, ``val`` must be ``None``. > > How is this different from an async send()? I see an asend() used in the example below but not anext(). Good catch! This is a typo, it should read ``agen.asend(val)``. Similarly to sync generators, where ?__next__()? call is equivalent of ?.send(None)?, ?__anext__()? awaitable is equivalent to ?.asend(None)' [..] > > 7. ``agen.ag_await``: The object that ``agen`` is currently awaiting on, > or ``None``. > > That's an interesting addition. I like it! There's no equivalent on normal generators, correct? We actually added that in 3.5 (last minute!). For sync generators the field is called ?.gi_yieldfrom?, for coroutines it?s ?.cr_await?, and for proposed async generators it will be ?.ag_await?. Thanks! Yury From brett at python.org Fri Jul 29 13:57:40 2016 From: brett at python.org (Brett Cannon) Date: Fri, 29 Jul 2016 17:57:40 +0000 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: On Fri, 29 Jul 2016 at 10:46 Yury Selivanov wrote: > Thanks a lot for the feedback, Brett! Comments inlined below: > > > On Jul 29, 2016, at 1:25 PM, Brett Cannon wrote: > > > [..] > > > > Performance is an additional point for this proposal: in our testing of > > the reference implementation, asynchronous generators are *2x* faster > > than an equivalent implemented as an asynchronous iterator. > > > > Another motivation is that types.coroutine becomes purely a > backwards-compatibility/low-level thing that is no longer required for > event loop frameworks to use on their generators which provide their async > API. So now an async framework can be written entirely in terms of def and > async def and not def and @types.coroutine. > > A slight misunderstanding here: @types.coroutine turns a normal old-style > generator into an awaitable object. That isn?t directly related to > asynchronous iteration. Frameworks like curio will continue using > @types.coroutine to implement future-like objects in event loops. > Ah, OK. So that would be yet another PEP to make that kind of change (and another keyword). > > [..] > > > > Support for Asynchronous Iteration Protocol > > ------------------------------------------- > > > > The protocol requires two special methods to be implemented: > > > > 1. An ``__aiter__`` method returning an *asynchronous iterator*. > > 2. An ``__anext__`` method returning an *awaitable* object, which uses > > ``StopIteration`` exception to "yield" values, and > > ``StopAsyncIteration`` exception to signal the end of the iteration. > > > > I assume this is in place of __iter__() and __next__(), i.e. an async > generator won't have both defined? > > Correct, async generators don?t have __iter__ and __next__. They can only > be iterated with an `async for`. > > [..] > > > > To solve this problem we propose to do the following: > > > > 1. Implement an ``aclose`` method on asynchronous generators > > returning a special *awaitable*. When awaited it > > throws a ``GeneratorExit`` into the suspended generator and > > iterates over it until either a ``GeneratorExit`` or > > a ``StopAsyncIteration`` occur. > > > > This is very similar to what the ``close()`` method does to regular > > Python generators, except that an event loop is required to execute > > ``aclose()``. > > > > I'm going to ask this now instead of later when there's more motivation > behind this question: do we need to append "a" to every async method we > have? If asynchronous generators won't have a normal close() then why can't > it just be close(), especially if people are not going to be calling it > directly and instead it will be event loops? I'm just leery of codifying > this practice of prepending "a" to every async method or function and > ending up in a future where I get tired of a specific letter of the > alphabet. > > I decided to use the prefix because we already use it in magic method > names: __anext__ and __aiter__. I think it also makes it easier to > understand the API of async generators (and understand how it?s different > from sync generators API). > > And while it?s entirely possible to drop the ?a? for async generator API, > it?s not so simple for other cases. Later, for Python 3.7, we might > consider adding ?aiter()? and ?anext()? builtins, for which we?d have to > use ?a? or ?async? prefix (we can?t reuse 'iter()' and 'next()? for async > generators). > I guess we just need to decide as a group that an 'a' prefix is what we want to signify something is asynchronous vs some other prefix like 'a_' or 'async', or 'async_' as people will follow this style choice in their own code going forward. > > [..] > > > > > 3. ``agen.anext(val)``: Returns an *awaitable*, that pushes the > > ``val`` object in the ``agen`` generator. When the ``agen`` has > > not yet been iterated, ``val`` must be ``None``. > > > > How is this different from an async send()? I see an asend() used in the > example below but not anext(). > > Good catch! This is a typo, it should read ``agen.asend(val)``. > > Similarly to sync generators, where ?__next__()? call is equivalent of > ?.send(None)?, ?__anext__()? awaitable is equivalent to ?.asend(None)' > > [..] > > > > 7. ``agen.ag_await``: The object that ``agen`` is currently awaiting on, > > or ``None``. > > > > That's an interesting addition. I like it! There's no equivalent on > normal generators, correct? > > We actually added that in 3.5 (last minute!). > > For sync generators the field is called ?.gi_yieldfrom?, for coroutines > it?s ?.cr_await?, and for proposed async generators it will be ?.ag_await?. > I would also clarify that "waiting on" means "what `await` has been called on (if anything as an `await` call might not have been used)" and not what the last yielded object happened to be (which is what my brain initially thought it was simply because the async generator is paused on the event loop returning based on what was yielded). > > Thanks! > Yury > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Fri Jul 29 14:07:45 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 29 Jul 2016 14:07:45 -0400 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: <2E871587-2DC0-488E-8223-B5B3309C710B@gmail.com> Few comments below: > On Jul 29, 2016, at 1:57 PM, Brett Cannon wrote: > > > > On Fri, 29 Jul 2016 at 10:46 Yury Selivanov wrote: > Thanks a lot for the feedback, Brett! Comments inlined below: > > > On Jul 29, 2016, at 1:25 PM, Brett Cannon wrote: > > > [..] > > > > Performance is an additional point for this proposal: in our testing of > > the reference implementation, asynchronous generators are *2x* faster > > than an equivalent implemented as an asynchronous iterator. > > > > Another motivation is that types.coroutine becomes purely a backwards-compatibility/low-level thing that is no longer required for event loop frameworks to use on their generators which provide their async API. So now an async framework can be written entirely in terms of def and async def and not def and @types.coroutine. > > A slight misunderstanding here: @types.coroutine turns a normal old-style generator into an awaitable object. That isn?t directly related to asynchronous iteration. Frameworks like curio will continue using @types.coroutine to implement future-like objects in event loops. > > Ah, OK. So that would be yet another PEP to make that kind of change (and another keyword). TBH I?m not really sure we need that. To separate coroutines from generators completely, you don?t just need another keyword ? you basically need to have a separate parallel implementation of the whole iteration protocol. I think @types.coroutine is a nice glue between two worlds, allowing us to reuse the code efficiently. Again, just my 2 cents. > > > > I'm going to ask this now instead of later when there's more motivation behind this question: do we need to append "a" to every async method we have? If asynchronous generators won't have a normal close() then why can't it just be close(), especially if people are not going to be calling it directly and instead it will be event loops? I'm just leery of codifying this practice of prepending "a" to every async method or function and ending up in a future where I get tired of a specific letter of the alphabet. > > I decided to use the prefix because we already use it in magic method names: __anext__ and __aiter__. I think it also makes it easier to understand the API of async generators (and understand how it?s different from sync generators API). > > And while it?s entirely possible to drop the ?a? for async generator API, it?s not so simple for other cases. Later, for Python 3.7, we might consider adding ?aiter()? and ?anext()? builtins, for which we?d have to use ?a? or ?async? prefix (we can?t reuse 'iter()' and 'next()? for async generators). > > I guess we just need to decide as a group that an 'a' prefix is what we want to signify something is asynchronous vs some other prefix like 'a_' or 'async', or 'async_' as people will follow this style choice in their own code going forward. I?m open to having this discussion. I don?t have a strong preference here; I, personally, like the ?a? prefix slightly better, because it?s consistent with __a*__ methods and easy to type. [..] > We actually added that in 3.5 (last minute!). > > For sync generators the field is called ?.gi_yieldfrom?, for coroutines it?s ?.cr_await?, and for proposed async generators it will be ?.ag_await?. > > I would also clarify that "waiting on" means "what `await` has been called on (if anything as an `await` call might not have been used)" and not what the last yielded object happened to be (which is what my brain initially thought it was simply because the async generator is paused on the event loop returning based on what was yielded). OK, I?ll try to clarify this! Yury From yarkot1 at gmail.com Fri Jul 29 14:20:41 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Fri, 29 Jul 2016 13:20:41 -0500 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: On Friday, July 29, 2016, Brett Cannon wrote: > > > On Fri, 29 Jul 2016 at 10:46 Yury Selivanov > wrote: > >> Thanks a lot for the feedback, Brett! Comments inlined below: >> >> > On Jul 29, 2016, at 1:25 PM, Brett Cannon > > wrote: >> > >> [..] >> > >> > Performance is an additional point for this proposal: in our testing of >> > the reference implementation, asynchronous generators are *2x* faster >> > than an equivalent implemented as an asynchronous iterator. >> > >> > Another motivation is that types.coroutine becomes purely a >> backwards-compatibility/low-level thing that is no longer required for >> event loop frameworks to use on their generators which provide their async >> API. So now an async framework can be written entirely in terms of def and >> async def and not def and @types.coroutine. >> >> A slight misunderstanding here: @types.coroutine turns a normal old-style >> generator into an awaitable object. That isn?t directly related to >> asynchronous iteration. Frameworks like curio will continue using >> @types.coroutine to implement future-like objects in event loops. >> > > Ah, OK. So that would be yet another PEP to make that kind of change (and > another keyword). > > >> >> [..] >> > >> > Support for Asynchronous Iteration Protocol >> > ------------------------------------------- >> > >> > The protocol requires two special methods to be implemented: >> > >> > 1. An ``__aiter__`` method returning an *asynchronous iterator*. >> > 2. An ``__anext__`` method returning an *awaitable* object, which uses >> > ``StopIteration`` exception to "yield" values, and >> > ``StopAsyncIteration`` exception to signal the end of the iteration. >> > >> > I assume this is in place of __iter__() and __next__(), i.e. an async >> generator won't have both defined? >> >> Correct, async generators don?t have __iter__ and __next__. They can only >> be iterated with an `async for`. >> >> [..] >> > >> > To solve this problem we propose to do the following: >> > >> > 1. Implement an ``aclose`` method on asynchronous generators >> > returning a special *awaitable*. When awaited it >> > throws a ``GeneratorExit`` into the suspended generator and >> > iterates over it until either a ``GeneratorExit`` or >> > a ``StopAsyncIteration`` occur. >> > >> > This is very similar to what the ``close()`` method does to regular >> > Python generators, except that an event loop is required to execute >> > ``aclose()``. >> > >> > I'm going to ask this now instead of later when there's more motivation >> behind this question: do we need to append "a" to every async method we >> have? If asynchronous generators won't have a normal close() then why can't >> it just be close(), especially if people are not going to be calling it >> directly and instead it will be event loops? I'm just leery of codifying >> this practice of prepending "a" to every async method or function and >> ending up in a future where I get tired of a specific letter of the >> alphabet. >> >> I decided to use the prefix because we already use it in magic method >> names: __anext__ and __aiter__. I think it also makes it easier to >> understand the API of async generators (and understand how it?s different >> from sync generators API). >> >> And while it?s entirely possible to drop the ?a? for async generator API, >> it?s not so simple for other cases. Later, for Python 3.7, we might >> consider adding ?aiter()? and ?anext()? builtins, for which we?d have to >> use ?a? or ?async? prefix (we can?t reuse 'iter()' and 'next()? for async >> generators). >> > > I guess we just need to decide as a group that an 'a' prefix is what we > want to signify something is asynchronous vs some other prefix like 'a_' or > 'async', or 'async_' as people will follow this style choice in their own > code going forward. > Hmm... I think we need to think about a future where, programmatically, there's little-to no distinction between async and synchronous functions. Pushing this down deeper in the system is the way to go. For one, it will serve simple multi-core use once gilectomy is completed (it, or something effectively equivalent will complete). For another, this is the path to reducing the functionally "useless" rewrite efforts of libraries (e.g. github.com/aio-libs), which somehow resemble all the efforts of migrating libraries from 2-3 (loosely). The resistance and unexpected time that 2-3 migration experienced won't readily be mimicked in async tasks - too much effort to get computer and I/O bound benefits? Maintain two versions of needed libraries, or jump languages is what will increasingly happen in the distributed (and more so IOT) world. Time to think about paving the way to async-as first class citizen world. That's probably too much for this PEP, but the topic (a- prefixing) is a good canary for the bigger picture we need to start mulling over. So in this context (and in general w/ async) asking the question "can we make it so it doesn't matter?" is a good one to always be asking - it will get is there. - Yarko > > >> >> [..] >> >> > >> > 3. ``agen.anext(val)``: Returns an *awaitable*, that pushes the >> > ``val`` object in the ``agen`` generator. When the ``agen`` has >> > not yet been iterated, ``val`` must be ``None``. >> > >> > How is this different from an async send()? I see an asend() used in >> the example below but not anext(). >> >> Good catch! This is a typo, it should read ``agen.asend(val)``. >> >> Similarly to sync generators, where ?__next__()? call is equivalent of >> ?.send(None)?, ?__anext__()? awaitable is equivalent to ?.asend(None)' >> >> [..] >> > >> > 7. ``agen.ag_await``: The object that ``agen`` is currently awaiting on, >> > or ``None``. >> > >> > That's an interesting addition. I like it! There's no equivalent on >> normal generators, correct? >> >> We actually added that in 3.5 (last minute!). >> >> For sync generators the field is called ?.gi_yieldfrom?, for coroutines >> it?s ?.cr_await?, and for proposed async generators it will be ?.ag_await?. >> > > I would also clarify that "waiting on" means "what `await` has been called > on (if anything as an `await` call might not have been used)" and not what > the last yielded object happened to be (which is what my brain initially > thought it was simply because the async generator is paused on the event > loop returning based on what was yielded). > > >> >> Thanks! >> Yury >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Fri Jul 29 14:40:55 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 29 Jul 2016 14:40:55 -0400 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Comments inlined: > On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak wrote: > > Hmm... I think we need to think about a future where, programmatically, there's little-to no distinction between async and synchronous functions. Pushing this down deeper in the system is the way to go. For one, it will serve simple multi-core use once gilectomy is completed (it, or something effectively equivalent will complete). For another, this is the path to reducing the functionally "useless" rewrite efforts of libraries (e.g. github.com/aio-libs), which somehow resemble all the efforts of migrating libraries from 2-3 (loosely). The resistance and unexpected time that 2-3 migration experienced won't readily be mimicked in async tasks - too much effort to get computer and I/O bound benefits? Maintain two versions of needed libraries, or jump languages is what will increasingly happen in the distributed (and more so IOT) world. When and *if* gilectomy is completed (or another project to remove the GIL), we will be able to do this: 1) Run existing async/await applications as is, but instead of running a process per core, we will be able to run a single process with many threads. Likely one asyncio (or other) event loop per thread. This is very speculative, but possible in theory. 2) Run existing blocking IO applications in several threads in one process. This is something that only sounds like an easy thing to do, I suspect that a lot of code will break (or dead lock) when the GIL is removed. Even if everything works perfectly well, threads aren?t answer to all problems ? try to manage 1000s of them. Long story short, even if we had no GIL at all, having async/await (and non-blocking IO) would make sense. And if you have async/await, with GIL or without, you will inevitably have different APIs and different IO low-level libs that drive them. There are ways to lessen the pain. For instance, I like Cory?s approach with hyper - implement protocols separately from IO, so that it?s easy to port them to various sync and async frameworks. > > Time to think about paving the way to async-as first class citizen world. > > That's probably too much for this PEP, but the topic (a- prefixing) is a good canary for the bigger picture we need to start mulling over. > > So in this context (and in general w/ async) asking the question "can we make it so it doesn't matter?" is a good one to always be asking - it will get is there. > Unfortunately there is no way to use the same APIs for both async/await and synchronous world. At least for CPython builtin types they have to have different names. I?m fine to discuss the ?a? prefix, but I?m a bit afraid that focusing on it too much will distract us from the PEP and details of it that really matter. Yury From yarkot1 at gmail.com Fri Jul 29 14:50:15 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Fri, 29 Jul 2016 13:50:15 -0500 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: On Friday, July 29, 2016, Yury Selivanov wrote: > Comments inlined: > > > > On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak > wrote: > > > > Hmm... I think we need to think about a future where, programmatically, > there's little-to no distinction between async and synchronous functions. > Pushing this down deeper in the system is the way to go. For one, it will > serve simple multi-core use once gilectomy is completed (it, or something > effectively equivalent will complete). For another, this is the path to > reducing the functionally "useless" rewrite efforts of libraries (e.g. > github.com/aio-libs), which somehow resemble all the efforts of migrating > libraries from 2-3 (loosely). The resistance and unexpected time that 2-3 > migration experienced won't readily be mimicked in async tasks - too much > effort to get computer and I/O bound benefits? Maintain two versions of > needed libraries, or jump languages is what will increasingly happen in the > distributed (and more so IOT) world. > > When and *if* gilectomy is completed (or another project to remove the > GIL), we will be able to do this: > > 1) Run existing async/await applications as is, but instead of running a > process per core, we will be able to run a single process with many > threads. Likely one asyncio (or other) event loop per thread. This is > very speculative, but possible in theory. > > 2) Run existing blocking IO applications in several threads in one > process. This is something that only sounds like an easy thing to do, I > suspect that a lot of code will break (or dead lock) when the GIL is > removed. Even if everything works perfectly well, threads aren?t answer to > all problems ? try to manage 1000s of them. > > Long story short, even if we had no GIL at all, having async/await (and > non-blocking IO) would make sense. And if you have async/await, with GIL > or without, you will inevitably have different APIs and different IO > low-level libs that drive them. > > There are ways to lessen the pain. For instance, I like Cory?s approach > with hyper - implement protocols separately from IO, so that it?s easy to > port them to various sync and async frameworks. > > > > > Time to think about paving the way to async-as first class citizen world. > > > > That's probably too much for this PEP, but the topic (a- prefixing) is a > good canary for the bigger picture we need to start mulling over. > > > > So in this context (and in general w/ async) asking the question "can we > make it so it doesn't matter?" is a good one to always be asking - it will > get is there. > > > > Unfortunately there is no way to use the same APIs for both async/await > and synchronous world. At least for CPython builtin types they have to > have different names. > > I?m fine to discuss the ?a? prefix, but I?m a bit afraid that focusing on > it too much will distract us from the PEP and details of it that really > matter. > > Yury To keep it simple, try thinking like this (and yes, Yury, apologies - this is now a side discussion, and not about this pep): everything in CPython is async, and if you don't want async, you don't need to know about, you run a single async task and don't need to know more... Can we get there? That would be cool... - Yarko -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Fri Jul 29 15:06:31 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 29 Jul 2016 19:06:31 +0000 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: I'm personally think that `a` prefix is pretty fine and understandable. Do we really need to implement full PEP-342 spec in async way? What are use cases? My first thought is PEP-255-like simple async generators should be enough. Full PEP-342 styled generators are clue for implementing things like twisted.inlineCallback and tornado.gen. Thanks to PEP-492 we don't need it anymore. All my requirements are covered perfectly fine by existing `__aiter__`/`__anext__`. What I really need is allowing to write just `yield val` from coroutine but never need for `resp = yield req` On Fri, Jul 29, 2016 at 8:50 PM Yarko Tymciurak wrote: > > > On Friday, July 29, 2016, Yury Selivanov wrote: > >> Comments inlined: >> >> >> > On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak wrote: >> > >> > Hmm... I think we need to think about a future where, >> programmatically, there's little-to no distinction between async and >> synchronous functions. Pushing this down deeper in the system is the way to >> go. For one, it will serve simple multi-core use once gilectomy is >> completed (it, or something effectively equivalent will complete). For >> another, this is the path to reducing the functionally "useless" rewrite >> efforts of libraries (e.g. github.com/aio-libs), which somehow resemble >> all the efforts of migrating libraries from 2-3 (loosely). The resistance >> and unexpected time that 2-3 migration experienced won't readily be >> mimicked in async tasks - too much effort to get computer and I/O bound >> benefits? Maintain two versions of needed libraries, or jump languages is >> what will increasingly happen in the distributed (and more so IOT) world. >> >> When and *if* gilectomy is completed (or another project to remove the >> GIL), we will be able to do this: >> >> 1) Run existing async/await applications as is, but instead of running a >> process per core, we will be able to run a single process with many >> threads. Likely one asyncio (or other) event loop per thread. This is >> very speculative, but possible in theory. >> >> 2) Run existing blocking IO applications in several threads in one >> process. This is something that only sounds like an easy thing to do, I >> suspect that a lot of code will break (or dead lock) when the GIL is >> removed. Even if everything works perfectly well, threads aren?t answer to >> all problems ? try to manage 1000s of them. >> >> Long story short, even if we had no GIL at all, having async/await (and >> non-blocking IO) would make sense. And if you have async/await, with GIL >> or without, you will inevitably have different APIs and different IO >> low-level libs that drive them. >> >> There are ways to lessen the pain. For instance, I like Cory?s approach >> with hyper - implement protocols separately from IO, so that it?s easy to >> port them to various sync and async frameworks. >> >> > >> > Time to think about paving the way to async-as first class citizen >> world. >> > >> > That's probably too much for this PEP, but the topic (a- prefixing) is >> a good canary for the bigger picture we need to start mulling over. >> > >> > So in this context (and in general w/ async) asking the question "can >> we make it so it doesn't matter?" is a good one to always be asking - it >> will get is there. >> > >> >> Unfortunately there is no way to use the same APIs for both async/await >> and synchronous world. At least for CPython builtin types they have to >> have different names. >> >> I?m fine to discuss the ?a? prefix, but I?m a bit afraid that focusing on >> it too much will distract us from the PEP and details of it that really >> matter. >> >> Yury > > > To keep it simple, try thinking like this (and yes, Yury, apologies - this > is now a side discussion, and not about this pep): everything in CPython > is async, and if you don't want async, you don't need to know about, you > run a single async task and don't need to know more... > > Can we get there? > That would be cool... > > - Yarko > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Fri Jul 29 15:07:16 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 29 Jul 2016 15:07:16 -0400 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: <56D00026-184B-4E08-A1D2-3B945EF7C477@gmail.com> > On Jul 29, 2016, at 2:50 PM, Yarko Tymciurak wrote: > To keep it simple, try thinking like this (and yes, Yury, apologies - this is now a side discussion, and not about this pep): everything in CPython is async, and if you don't want async, you don't need to know about, you run a single async task and don't need to know more... > > Can we get there? > That would be cool... So something like what they have in Golang? I don?t know if that?s possible in CPython... Yury From yselivanov at gmail.com Fri Jul 29 15:14:36 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 29 Jul 2016 15:14:36 -0400 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: <8D03898A-2B13-4916-A367-7B1D74B64A4C@gmail.com> > On Jul 29, 2016, at 3:06 PM, Andrew Svetlov wrote: > > I'm personally think that `a` prefix is pretty fine and understandable. > > Do we really need to implement full PEP-342 spec in async way? > What are use cases? > My first thought is PEP-255-like simple async generators should be enough. > > Full PEP-342 styled generators are clue for implementing things like twisted.inlineCallback and tornado.gen. > > Thanks to PEP-492 we don't need it anymore. > > All my requirements are covered perfectly fine by existing `__aiter__`/`__anext__`. > What I really need is allowing to write just `yield val` from coroutine but never need for `resp = yield req` There are a few reasons for having asend() and athrow(): 1) They will have to be implemented regardless; aclose() is a slightly limited version of athrow(), and __anext__() is almost the same thing as asend(). It?s 5-10 additional lines of code to have them. 2) __anext__() has to be a generator-like object, so that YIELD_FROM opcode works correctly with it. That means it has to implement ?send()? and ?throw()? methods. Which, in turn, means that even if we don?t expose ?agen.athrow()?, people would be able to do ?agen.__anext__().throw()?. But that?s a very error-prone thing to do. 3) Having ?anext()? and ?athrow()? enable you to implement decorators like `@contextlib.contextmanager` but for ?async with?. And I?m pretty sure that people will come up with even more creative things. To conclude, having ?asend()? and ?athrow()? doesn?t complicate the implementation at all, but makes asynchronous generators consistent and more compatible with synchronous generators, which I don?t think is a bad thing. Yury From andrew.svetlov at gmail.com Fri Jul 29 15:17:18 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 29 Jul 2016 19:17:18 +0000 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: <8D03898A-2B13-4916-A367-7B1D74B64A4C@gmail.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <8D03898A-2B13-4916-A367-7B1D74B64A4C@gmail.com> Message-ID: Ok, if the implementation gets `asend()` etc. "for free" I support it. On Fri, Jul 29, 2016 at 9:14 PM Yury Selivanov wrote: > > > On Jul 29, 2016, at 3:06 PM, Andrew Svetlov > wrote: > > > > I'm personally think that `a` prefix is pretty fine and understandable. > > > > Do we really need to implement full PEP-342 spec in async way? > > What are use cases? > > My first thought is PEP-255-like simple async generators should be > enough. > > > > Full PEP-342 styled generators are clue for implementing things like > twisted.inlineCallback and tornado.gen. > > > > Thanks to PEP-492 we don't need it anymore. > > > > All my requirements are covered perfectly fine by existing > `__aiter__`/`__anext__`. > > What I really need is allowing to write just `yield val` from coroutine > but never need for `resp = yield req` > > There are a few reasons for having asend() and athrow(): > > 1) They will have to be implemented regardless; aclose() is a slightly > limited version of athrow(), and __anext__() is almost the same thing as > asend(). It?s 5-10 additional lines of code to have them. > > 2) __anext__() has to be a generator-like object, so that YIELD_FROM > opcode works correctly with it. That means it has to implement ?send()? and > ?throw()? methods. Which, in turn, means that even if we don?t expose > ?agen.athrow()?, people would be able to do ?agen.__anext__().throw()?. > But that?s a very error-prone thing to do. > > 3) Having ?anext()? and ?athrow()? enable you to implement decorators like > `@contextlib.contextmanager` but for ?async with?. And I?m pretty sure > that people will come up with even more creative things. > > To conclude, having ?asend()? and ?athrow()? doesn?t complicate the > implementation at all, but makes asynchronous generators consistent and > more compatible with synchronous generators, which I don?t think is a bad > thing. > > Yury -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gerasimov-M-N at yandex.ru Sat Jul 30 06:32:46 2016 From: Gerasimov-M-N at yandex.ru (=?UTF-8?B?0JPQtdGA0LDRgdC40LzQvtCyINCc0LjRhdCw0LjQuw==?=) Date: Sat, 30 Jul 2016 13:32:46 +0300 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: > When an asynchronous generator is about to be garbage collected, > it calls its cached finalizer. The assumption is that the finalizer > will schedule an ``aclose()`` call with the loop that was active > when the iteration started. > > For instance, here is how asyncio can be modified to allow > safe finalization of asynchronous generators:: > > # asyncio/base_events.py > > class BaseEventLoop: > > def run_forever(self): > ... > old_finalizer = sys.get_asyncgen_finalizer() > sys.set_asyncgen_finalizer(self._finalize_asyncgen) > try: > ... > finally: > sys.set_asyncgen_finalizer(old_finalizer) > ... > > def _finalize_asyncgen(self, gen): > self.create_task(gen.aclose()) > > ``sys.set_asyncgen_finalizer`` is thread-specific, so several event > loops running in parallel threads can use it safely. When asynchronous generator is about to be garbage collected event loop starts task to execute aclose, right? Can't such situation happen when this task is not finished at the moment event loop is closed? Something like: async def gen(): try: yield 1 yield 2 yield 3 except GeneratorExit as exc: await asyncio.sleep(100) raise exc async def main(): async for i in gen(): if i == 1: break # main() and event loop is about to finish here, # while task to aclose will last much longer. if __name__ == "__main__": loop = asyncio.get_event_loop() loop.run_until_complete(main()) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Sat Jul 30 06:57:45 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Sat, 30 Jul 2016 03:57:45 -0700 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> Message-ID: <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> > On Jul 30, 2016, at 3:32 AM, ????????? ?????? wrote: > > When asynchronous generator is about to be garbage collected event loop starts task to execute aclose, right? Can't such situation happen when this task is not finished at the moment event loop is closed? Something like: Yes, such situations can happen. BUT they can happen for regular coroutines too! Try running the following: async def coro1(): try: print('try') await asyncio.sleep(1) finally: await asyncio.sleep(0) print('finally') async def coro2(): await asyncio.sleep(0) loop = asyncio.get_event_loop() loop.create_task(coro1()) loop.run_until_complete(coro2()) loop.close() In the above script, coro1() enters its `try` block and prints ?try?; loop ends the execution because coro2() is completed; and coro1() will never execute its `finally` block. Long story short - you have to be extra careful when you?re closing the event loop. I believe that not executing ?finally? statements when the process is about to complete will be more common for coroutines than async generators. Also, in real life programs you don?t run and close event loops more than once per process lifetime. So it should be fine if you have a few non-exhausted generators and coroutines being GCed without proper finalization just before the process exits. And, BTW, Python interpreter will issue a ResourceWarning that it cannot finalize coroutines and/or async generators. Yury From Gerasimov-M-N at yandex.ru Sat Jul 30 08:01:57 2016 From: Gerasimov-M-N at yandex.ru (=?UTF-8?B?0JPQtdGA0LDRgdC40LzQvtCyINCc0LjRhdCw0LjQuw==?=) Date: Sat, 30 Jul 2016 15:01:57 +0300 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> Message-ID: Looks like to be sure everything finished correctly, we'll need to do something like this every time: loop.run_until_complete(main()) pending = asyncio.Task.all_tasks() loop.run_until_complete(asyncio.gather(*pending)) loop.close() Bad thing here I think is that we can't split manually created tasks (wich warnings we might would like to get to complete this tasks in other places) and AG's close tasks wich we can't complete other place then here. Don't you like idea to await all AG's close tasks done right before event loop is closed? I mean to modify event loop this way: # asyncio/base_events.py class BaseEventLoop: def _finalize_asyncgen(self, gen): task = self.create_task(gen.aclose()) self._close_tasks.append(task) def close(): self.run_until_complete( asyncio.gather(*self._close_tasks) ) ... In this case user would be able to use async generators without worring he will get warning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sat Jul 30 08:06:31 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 30 Jul 2016 12:06:31 +0000 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> Message-ID: It may wait forever in case of malformed generator implementation. On Sat, Jul 30, 2016 at 2:04 PM ????????? ?????? wrote: > Looks like to be sure everything finished correctly, we'll need to do > something like this every time: > > loop.run_until_complete(main()) > > pending = asyncio.Task.all_tasks() > loop.run_until_complete(asyncio.gather(*pending)) > > loop.close() > > Bad thing here I think is that we can't split manually created tasks (wich > warnings we might would like to get to complete this tasks in other places) > and AG's close tasks wich we can't complete other place then here. > > Don't you like idea to await all AG's close tasks done right before event > loop is closed? I mean to modify event loop this way: > > # asyncio/base_events.py > > class BaseEventLoop: > def _finalize_asyncgen(self, gen): > task = self.create_task(gen.aclose()) > self._close_tasks.append(task) > > def close(): > self.run_until_complete( > asyncio.gather(*self._close_tasks) > ) > ... > > In this case user would be able to use async generators without worring he > will get warning. > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gerasimov-M-N at yandex.ru Sat Jul 30 08:15:31 2016 From: Gerasimov-M-N at yandex.ru (=?UTF-8?B?0JPQtdGA0LDRgdC40LzQvtCyINCc0LjRhdCw0LjQuw==?=) Date: Sat, 30 Jul 2016 15:15:31 +0300 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> Message-ID: <827f4f61-e5a0-6c63-99a1-1e1820381dfe@yandex.ru> > It may wait forever in case of malformed generator implementation. > Yes, but every "await task" can wait forever in case of malformed task implementation. We can't do anything in case user will write: while True: pass -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sat Jul 30 09:49:37 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 30 Jul 2016 13:49:37 +0000 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: <827f4f61-e5a0-6c63-99a1-1e1820381dfe@yandex.ru> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> <827f4f61-e5a0-6c63-99a1-1e1820381dfe@yandex.ru> Message-ID: When you have `await task` line you probably know what task it is, you can debug and fix it. But hanging on `loop.close()` because some unknown task hangs looks very confusing. The approach gives no clue for what task is malformed. Users will blame that `loop.close()` hangs forever without any reason or additional information. On Sat, Jul 30, 2016 at 2:15 PM ????????? ?????? wrote: > > It may wait forever in case of malformed generator implementation. > > > Yes, but every "await task" can wait forever in case of malformed task > implementation. We can't do anything in case user will write: > > while True: > pass > > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gerasimov-M-N at yandex.ru Sat Jul 30 11:08:47 2016 From: Gerasimov-M-N at yandex.ru (=?UTF-8?B?0JPQtdGA0LDRgdC40LzQvtCyINCc0LjRhdCw0LjQuw==?=) Date: Sat, 30 Jul 2016 18:08:47 +0300 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> <827f4f61-e5a0-6c63-99a1-1e1820381dfe@yandex.ru> Message-ID: <09b968cf-8e81-db6f-f4a5-7b1e5d2c5af9@yandex.ru> > When you have `await task` line you probably know what task it is, you > can debug and fix it. > > But hanging on `loop.close()` because some unknown task hangs looks > very confusing. > The approach gives no clue for what task is malformed. > > Users will blame that `loop.close()` hangs forever without any reason > or additional information. ok, regardless placing this code inside `loop.close()` is there any alternative to sure async generators inside `main()` are closed? loop.run_until_complete(main()) pending = asyncio.Task.all_tasks() loop.run_until_complete(asyncio.gather(*pending)) loop.close() Currently, I just don't see any other way to use some async generator (that needs some time to be closed) without getting warning. In case malformed generator is unwantedbehaviour may be it'll be ok to show warning in debug mode if some close task isn't finished after some timeout? -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Sat Jul 30 11:39:40 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 30 Jul 2016 15:39:40 +0000 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: <09b968cf-8e81-db6f-f4a5-7b1e5d2c5af9@yandex.ru> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> <827f4f61-e5a0-6c63-99a1-1e1820381dfe@yandex.ru> <09b968cf-8e81-db6f-f4a5-7b1e5d2c5af9@yandex.ru> Message-ID: Adding the feature under PYTHONASYNCIODEBUG flag sounds perfectly reasonable for me. But not in production mode. On Sat, Jul 30, 2016 at 6:08 PM ????????? ?????? wrote: > > When you have `await task` line you probably know what task it is, you can > debug and fix it. > > But hanging on `loop.close()` because some unknown task hangs looks very > confusing. > The approach gives no clue for what task is malformed. > > Users will blame that `loop.close()` hangs forever without any reason or > additional information. > > > ok, regardless placing this code inside `loop.close()` is there any > alternative to sure async generators inside `main()` are closed? > > > loop.run_until_complete(main()) > > pending = asyncio.Task.all_tasks() > loop.run_until_complete(asyncio.gather(*pending)) > > loop.close() > > Currently, I just don't see any other way to use some async generator > (that needs some time to be closed) without getting warning. > > In case malformed generator is unwanted behaviour may be it'll be ok to > show warning in debug mode if some close task isn't finished after some > timeout? > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Sat Jul 30 15:24:17 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Sat, 30 Jul 2016 12:24:17 -0700 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> <827f4f61-e5a0-6c63-99a1-1e1820381dfe@yandex.ru> <09b968cf-8e81-db6f-f4a5-7b1e5d2c5af9@yandex.ru> Message-ID: [..] > Currently, I just don't see any other way to use some async generator (that needs some time to be closed) without getting warning. It will be a very rare case ? you need a generator that produces an infinite series of values with a try..finally block. For those kinds of generators you will have a standard asyncio warning that a task was GCed in pending state. We can maintain a weak set of ?aclose? tasks in the event loop, and make it a public and documented property. That way you?ll be able to call gather() with a timeout on that list before closing the loop. The PEP doesn?t specify how exactly asyncio should be modified. This is something we can discuss when the PEP patch is reviewed. Yury From Gerasimov-M-N at yandex.ru Sun Jul 31 06:23:31 2016 From: Gerasimov-M-N at yandex.ru (=?UTF-8?B?0JPQtdGA0LDRgdC40LzQvtCyINCc0LjRhdCw0LjQuw==?=) Date: Sun, 31 Jul 2016 13:23:31 +0300 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <025F6E05-2351-42B9-AC1B-994B6BD799F1@gmail.com> <827f4f61-e5a0-6c63-99a1-1e1820381dfe@yandex.ru> <09b968cf-8e81-db6f-f4a5-7b1e5d2c5af9@yandex.ru> Message-ID: <7ddb0d43-9997-5b12-a8a1-a738d691179e@yandex.ru> > It will be a very rare case ? you need a generator that produces an infinite series of values with a try..finally block. If I'm not mistaken it'll be any generator with `break` before it yieldedlast value(and with try...finally block that is usually context manager). def gen(): async with cm_long_aexit(): yield 1 yield 2 yield 3 async for i in gen(): if i == 2: break > We can maintain a weak set of ?aclose? tasks in the event loop, and make it a public and documented property. That way you?ll be able to call gather() with a timeout on that list before closing the loop. It solves the issue. -------------- next part -------------- An HTML attachment was scrubbed... URL: