From flagzeta at gmail.com Tue Nov 1 12:15:33 2016 From: flagzeta at gmail.com (Federico Marani) Date: Tue, 1 Nov 2016 16:15:33 +0000 Subject: [Async-sig] async contextmanager Message-ID: Has anyone tried to build an asynchronous version of @contextmanager? Is it even possible... given the use of yield? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Tue Nov 1 13:11:30 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Tue, 1 Nov 2016 19:11:30 +0200 Subject: [Async-sig] async contextmanager In-Reply-To: References: Message-ID: Certainly. You can use asyncio_extras for that: http://pythonhosted.org/asyncio_extras/#asyncio-extras-contextmanager 01.11.2016, 18:15, Federico Marani kirjoitti: > Has anyone tried to build an asynchronous version of @contextmanager? > Is it even possible... given the use of yield? > > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Gerasimov-M-N at yandex.ru Tue Nov 1 13:23:51 2016 From: Gerasimov-M-N at yandex.ru (=?UTF-8?B?0JPQtdGA0LDRgdC40LzQvtCyINCc0LjRhdCw0LjQuw==?=) Date: Tue, 1 Nov 2016 20:23:51 +0300 Subject: [Async-sig] async contextmanager In-Reply-To: References: Message-ID: <057ff8ae-444d-66c1-064f-4c19bb6f88dc@yandex.ru> I played with it some time ago. Actually, original code with few changes works fine. Here's example: https://gist.github.com/germn/6b2f8a1afbfc322daa01365cacd1d9ad Test it: import asyncio from acontextlib import acontextmanager state = [] @acontextmanager async def woohoo(): state.append(1) yield 42 state.append(999) async def main(): async with woohoo() as x: assert state == [1] assert x == 42 if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(main()) 01.11.2016 19:15, Federico Marani ?????: > Has anyone tried to build an asynchronous version of @contextmanager? > Is it even possible... given the use of yield? > > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vxgmichel at gmail.com Tue Nov 1 13:35:35 2016 From: vxgmichel at gmail.com (Vincent Michel) Date: Tue, 1 Nov 2016 18:35:35 +0100 Subject: [Async-sig] async contextmanager In-Reply-To: References: Message-ID: <5818D267.309@gmail.com> > Has anyone tried to build an asynchronous version of @contextmanager? Is > it even possible... given the use of yield? Thanks to PEP 525 - Asynchronous generators [1], it will be possible in python 3.6. Here's a possible implementation [2]. [1] https://www.python.org/dev/peps/pep-0525/ [2] https://github.com/vxgmichel/aiostream/blob/master/aiostream/context_utils.py /Vincent From gmludo at gmail.com Tue Nov 1 20:18:33 2016 From: gmludo at gmail.com (Ludovic Gasc) Date: Wed, 2 Nov 2016 01:18:33 +0100 Subject: [Async-sig] Status of asyncio.readthedocs.io ? In-Reply-To: References: Message-ID: Hi, FYI, I've did a workshop with 35 attendees during the Pycon-FR 2016 about AsyncIO and aiohttp.web. I used mainly http://asyncio.readthedocs.io/ as course support. I'm not a professional trainer, nevertheless, everybody seem to be happy. My main advice is to ask at the beginning of the session the level of each attendee, and group by level. Because, when you take a random Python developer in community, you can have somebody who has already play with the async pattern and AsyncIO, like (s)he has never read a piece of documentation about async pattern. You can't have the same approach for both profiles. Ideally, to be 2 or 3 is a good idea, to animate each group. The list of levels I've used: 1. I never use Python 2. I've rewritten Python scripts 3. I've already use Flask/Django 4. I've did async without AsyncIO 5. I've some troubles with AsyncIO 6. I'm here only to troll about AsyncIO The latest group is more important than you think: It helps you to add some humor, and also to split advanced people with you can easily debate without to "pollute" the other groups who only want to learn AsyncIO and aiohttp.web. Thanks for people who has assisted me during the workshop. Have a nice day. PS: If you have some tips and tricks to animate AsyncIO workshops, I'm interested in. -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ 2016-10-14 17:12 GMT+02:00 Ludovic Gasc : > Hi everybody, > > For me, http://asyncio.readthedocs.io/ is enough stable now to promote > the project. > Ok for everybody, or you want to add content or change the structure ? > > BTW, thanks to all contributors, especially Victor Stinner and Mike M?ller > from Python-Academy: > https://github.com/asyncio-doc/asyncio-doc/graphs/contributors > > Have a nice week. > > PS: No answer before Sunday = approved by people in the mailing-lists ;-) > -- > Ludovic Gasc (GMLudo) > http://www.gmludo.eu/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Nov 5 20:09:36 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 5 Nov 2016 17:09:36 -0700 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world Message-ID: I just posted a long blog/essay that's probably of interest to folks here: https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/ The short version: I think curio something important to teach us; I tried to figure out what that is and how we can learn from it. -n -- Nathaniel J. Smith -- https://vorpus.org From cory at lukasa.co.uk Sun Nov 6 05:55:25 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Sun, 6 Nov 2016 10:55:25 +0000 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: Message-ID: > On 6 Nov 2016, at 00:09, Nathaniel Smith wrote: > > I just posted a long blog/essay that's probably of interest to folks here: > > https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/ > > The short version: I think curio something important to teach us; I > tried to figure out what that is and how we can learn from it. This is a great post Nathaniel, and I think there?s a lot of value to extract from this for everyone. In the short term, I?m going to address Twisted?s issue because it seems like the most glaring ?this is a stupid bug? problem, and a quick glance at the relevant interfaces suggests it?s easily resolved. It also undermines my own personal mission to make Twisted a great HTTP/2 server. ;) Cory From brett at python.org Sun Nov 6 23:20:20 2016 From: brett at python.org (Brett Cannon) Date: Mon, 07 Nov 2016 04:20:20 +0000 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: Message-ID: I'll also reiterate Cory's compliments that the post was great! For me there are two questions the post raises. One is how do we keep people from tying themselves to any one event loop? I view sans-io as the start of this, but it does require getting people to know about it, implement protocols that way, and then continuing to abstract themselves in such a way so as to not tie themselves down. But then we start to go up a level to things that work at the HTTP level and then it starts to get complicated and I don't know if we have a solution yet for e.g. an async GitHub SDK library that is event loop-agnostic when it comes to HTTP requests/responses. Two, how long do we put off "the future" for some async/await-native event loop to emerge and hit production quality? I totally understand David wanting to keep curio experimental, but at some point something will either need to reach stable for people to seriously use or "the future" will simply stay in the future and people will just tie themselves to the asyncio abstractions, making it that much harder for people to try something else. As I think all of us realize, async/await is giving us a rare opportunity to set the future tone for networking code in Python, but at some point a tipping point will be reached and whatever common practice is in place at that point might calcify, so figuring out how we want things to be in the general landscape would be good. I would also like to thank everyone involved with defining async/await. I think the fact that async/await is an API and not something tightly bound to any specific event loop like in pretty much every other async-supporting language has been very beneficial to us and something to celebrate. The sheer fact that people who don't like asyncio, Twisted, Tornado, or curio have other options is fantastic. On Sun, 6 Nov 2016 at 02:55 Cory Benfield wrote: > > > On 6 Nov 2016, at 00:09, Nathaniel Smith wrote: > > > > I just posted a long blog/essay that's probably of interest to folks > here: > > > > > https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/ > > > > The short version: I think curio something important to teach us; I > > tried to figure out what that is and how we can learn from it. > > This is a great post Nathaniel, and I think there?s a lot of value to > extract from this for everyone. > > In the short term, I?m going to address Twisted?s issue because it seems > like the most glaring ?this is a stupid bug? problem, and a quick glance at > the relevant interfaces suggests it?s easily resolved. It also undermines > my own personal mission to make Twisted a great HTTP/2 server. ;) > > Cory > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Nov 7 01:41:10 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 6 Nov 2016 22:41:10 -0800 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: Message-ID: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> > On Nov 6, 2016, at 8:20 PM, Brett Cannon wrote: > > For me there are two questions the post raises. One is how do we keep people from tying themselves to any one event loop? Deprecate, then remove, get_event_loop() :-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at dabeaz.com Mon Nov 7 04:46:16 2016 From: dave at dabeaz.com (David Beazley) Date: Mon, 7 Nov 2016 03:46:16 -0600 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: Message-ID: I don't know if curio (or something like it) is the future or not, but it is something that I'm building for myself so that I can use it. I like it. It fits my brain. I'd just like to reiterate Brett's comment about async/await being a protocol in Python and something that can be customized. I don't know if that was by design or a happy accident, but so far as I can tell, it might be unique to Python. Every other language that has this seems to have it pretty well locked down to a very specific runtime implementation involving callbacks and futures. Python's approach allowed me to run with it in a completely different direction. I think that's pretty neat. Cheers, Dave > I would also like to thank everyone involved with defining async/await. I think the fact that async/await is an API and not something tightly bound to any specific event loop like in pretty much every other async-supporting language has been very beneficial to us and something to celebrate. The sheer fact that people who don't like asyncio, Twisted, Tornado, or curio have other options is fantastic. > > On Sun, 6 Nov 2016 at 02:55 Cory Benfield > wrote: > > > On 6 Nov 2016, at 00:09, Nathaniel Smith > wrote: > > > > I just posted a long blog/essay that's probably of interest to folks here: > > > > https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/ > > > > The short version: I think curio something important to teach us; I > > tried to figure out what that is and how we can learn from it. > > This is a great post Nathaniel, and I think there?s a lot of value to extract from this for everyone. > > In the short term, I?m going to address Twisted?s issue because it seems like the most glaring ?this is a stupid bug? problem, and a quick glance at the relevant interfaces suggests it?s easily resolved. It also undermines my own personal mission to make Twisted a great HTTP/2 server. ;) > > Cory > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vxgmichel at gmail.com Mon Nov 7 07:51:58 2016 From: vxgmichel at gmail.com (Vincent Michel) Date: Mon, 7 Nov 2016 13:51:58 +0100 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: Message-ID: <2bc62330-a547-ec3e-2b60-3b718ab48426@gmail.com> Thanks Nathaniel for the great post! It's indeed very impressive to see how curio worked his way around problems that I thought were part of any async library, by simply giving up on callbacks. One aspect of curio that I find particularly interesting is how it hides the event loop (or kernel) by: 1 - Using "yield (TRAP, *args)" as the main way to communicate with the event loop (no need to have it as a reference). 2 - Exposing a run function that starts the loop and make sure it safely terminates. Even though there's a long way to go before we can have point 1 in asyncio (or at least a compatibility layer), I think point 2 is easy to implement and could bring something valuable. So here's a proposal: Add an asyncio.run function =========================== ... and promote it as the standard to run asynchronous applications. Implementation -------------- It could roughly be implemented as: ``` def run(main_coro, *, loop=None): if loop is not None: loop = asyncio.get_event_loop() try: return loop.run_until_complete(main_coro) finally: # More clean-up here? loop.close() ``` Example ------- Instead of writing: ``` loop = asyncio.get_event_loop() queue = asyncio.Queue(loop=loop) producer_coro = produce(queue, 10) consumer_coro = consume(queue) gather = asyncio.gather(producer_coro, consumer_coro, loop=loop) loop.run_until_complete(gather) loop.close() ``` We could promote the following structure: ``` async def main(): queue = asyncio.Queue() producer_coro = produce(queue, 10) consumer_coro = consume(queue) await asyncio.gather(producer_coro, consumer_coro) if __name__ == '__main__': asyncio.run(main()) ``` What do we get from that? ------------------------- - A clear separation between the synchronous and the asynchronous world. Asynchronous objects should only be created inside an asynchronous context. - No explicit vs implicit loop issues, PR #452 guarantees that objects created inside coroutines and callbacks will get the right event loop (so loop references can be omitted everywhere). - The event loop disappears completely from the user code and becomes a low-level detail. - It provides a proper way to clean things up after running the loop (e.g make sure all the pending callbacks are executed before the loop is closed, or maybe a curio-like behavior to wait for all "non-daemonic" tasks to complete.) Any limitations? ---------------- One issue I can think of is the handling of KeyboardInterrupt. For instance, how to transform the TCP server example from the docs to fit the new standard? Ideally, we should be able to write something like this: ``` async def main(): server = await asyncio.start_server(handle_echo, '127.0.0.1', 8888) print('Serving on {}'.format(server.sockets[0].getsockname())) await asyncio.wait_for_interrupt() server.close() await server.wait_closed()) ``` But the handling of interrupts is not completely settled yet (see PR #305 and issue #341). Also, some asyncio-based library might already implement a similar but specific run function. Could there be a conflict here? Related topics -------------- - [Explicit vs Implicit event loop discussion on python-tulip][1] - [asyncio PR #452: Make get_event_loop() return the current loop if called from coroutines/callbacks][2] [1]: https://groups.google.com/forum/#!topic/python-tulip/yF9C-rFpiKk [2]: https://github.com/python/asyncio/pull/452 I hope it makes sense. /Vincent From brett at python.org Mon Nov 7 12:50:50 2016 From: brett at python.org (Brett Cannon) Date: Mon, 07 Nov 2016 17:50:50 +0000 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> Message-ID: On Sun, 6 Nov 2016 at 22:41 Glyph Lefkowitz wrote: > > On Nov 6, 2016, at 8:20 PM, Brett Cannon wrote: > > For me there are two questions the post raises. One is how do we keep > people from tying themselves to any one event loop? > > > Deprecate, then remove, get_event_loop() :-). > Is there a bug filed for that at https://github.com/python/asyncio? -Brett > > -glyph > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Mon Nov 7 12:56:53 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Mon, 7 Nov 2016 12:56:53 -0500 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> Message-ID: <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> > On Nov 7, 2016, at 12:50 PM, Brett Cannon wrote: > > > > On Sun, 6 Nov 2016 at 22:41 Glyph Lefkowitz wrote: > >> On Nov 6, 2016, at 8:20 PM, Brett Cannon wrote: >> >> For me there are two questions the post raises. One is how do we keep people from tying themselves to any one event loop? > > Deprecate, then remove, get_event_loop() :-). > > Is there a bug filed for that at https://github.com/python/asyncio? I don?t think we need to deprecate get_event_loop(). With https://github.com/python/asyncio/pull/452 merged in 3.6, get_event_loop becomes more predictable. Now it?s a documentation issue (I?m trying to work on that) to explain asyncio users not to use it (and where they *do* need to use it). I will also open a PR soon to add asyncio.main() function (or asyncio.run()) to further simplify working with asyncio & its documentation. That should make get_event_loop to disappear for end users. Yury From glyph at twistedmatrix.com Mon Nov 7 13:38:16 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 7 Nov 2016 10:38:16 -0800 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> Message-ID: <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> > On Nov 7, 2016, at 9:56 AM, Yury Selivanov wrote: > > >> On Nov 7, 2016, at 12:50 PM, Brett Cannon wrote: >> >> >> >> On Sun, 6 Nov 2016 at 22:41 Glyph Lefkowitz wrote: >> >>> On Nov 6, 2016, at 8:20 PM, Brett Cannon wrote: >>> >>> For me there are two questions the post raises. One is how do we keep people from tying themselves to any one event loop? >> >> Deprecate, then remove, get_event_loop() :-). >> >> Is there a bug filed for that at https://github.com/python/asyncio? > > I don?t think we need to deprecate get_event_loop(). With https://github.com/python/asyncio/pull/452 merged in 3.6, get_event_loop becomes more predictable. > > Now it?s a documentation issue (I?m trying to work on that) to explain asyncio users not to use it (and where they *do* need to use it). > > I will also open a PR soon to add asyncio.main() function (or asyncio.run()) to further simplify working with asyncio & its documentation. That should make get_event_loop to disappear for end users. Sorry, this was a bit tongue in cheek. This was something I said to Guido at the *very* beginning of Tulip development, when asked about mistakes Twisted has made: "don't have a global event loop, you'll never get away from it". I still think getting rid of a global loop would always be an improvement, although I suspect it's too late at this point. `await current_event_loop()` might make more sense in Asyncio as that's not really "global", similar to Curio's trap of the same design; however, I assume that this was an intentional design disagreement for a reason and I don't see that reason as having changed (as Yury indicates). -glyph From yselivanov at gmail.com Mon Nov 7 14:08:13 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Mon, 7 Nov 2016 14:08:13 -0500 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> Message-ID: <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> [..] > Sorry, this was a bit tongue in cheek. This was something I said to Guido at the *very* beginning of Tulip development, when asked about mistakes Twisted has made: "don't have a global event loop, you'll never get away from it". > > I still think getting rid of a global loop would always be an improvement, although I suspect it's too late at this point. `await current_event_loop()` might make more sense in Asyncio as that's not really "global", similar to Curio's trap of the same design; however, I assume that this was an intentional design disagreement for a reason and I don't see that reason as having changed (as Yury indicates). The latest update of get_event_loop is a step in the right direction. At least now we can document the best practices: 1. Have one ?main? coroutine to bootstrap/run your program; 2. Don?t design APIs that accept the loop parameter; instead design coroutine-first APIs and use get_event_loop in your library if you absolutely need the loop. 3. I want to add ?asyncio.main(coro)? function, which would create the loop, run the ?coro? coroutine, and correctly clean everything up. What you propose, IIUC is a step further: * Deprecate get_event_loop(); * Add ?current_event_loop()? coroutine. This will enforce (1) and (2), making asyncio library devs/users to focus more on coroutines and async/await. Am I understanding this all correctly? Yury From glyph at twistedmatrix.com Mon Nov 7 14:41:58 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 7 Nov 2016 11:41:58 -0800 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> Message-ID: <07F33DBD-0EFB-458E-BD69-8476DD4886E3@twistedmatrix.com> > On Nov 7, 2016, at 11:08 AM, Yury Selivanov wrote: > > [..] >> Sorry, this was a bit tongue in cheek. This was something I said to Guido at the *very* beginning of Tulip development, when asked about mistakes Twisted has made: "don't have a global event loop, you'll never get away from it". >> >> I still think getting rid of a global loop would always be an improvement, although I suspect it's too late at this point. `await current_event_loop()` might make more sense in Asyncio as that's not really "global", similar to Curio's trap of the same design; however, I assume that this was an intentional design disagreement for a reason and I don't see that reason as having changed (as Yury indicates). > > The latest update of get_event_loop is a step in the right direction. At least now we can document the best practices: > > 1. Have one ?main? coroutine to bootstrap/run your program; > > 2. Don?t design APIs that accept the loop parameter; instead design coroutine-first APIs and use get_event_loop in your library if you absolutely need the loop. > > 3. I want to add ?asyncio.main(coro)? function, which would create the loop, run the ?coro? coroutine, and correctly clean everything up. > > What you propose, IIUC is a step further: > > * Deprecate get_event_loop(); > > * Add ?current_event_loop()? coroutine. > > This will enforce (1) and (2), making asyncio library devs/users to focus more on coroutines and async/await. > > Am I understanding this all correctly? Yep. It's not so much making users focus more on coroutines, as having a way to pass a loop to a coroutine that is explicit (the coro needs to be scheduled on a loop already, so the binding has been explicitly specified) but unobtrusive. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Mon Nov 7 14:48:46 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Mon, 07 Nov 2016 19:48:46 +0000 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> Message-ID: Maybe we need couple other coroutines: 1. asyncio.run_in_executor() 2. asyncio.create_task(). Yes, ensure_future() does the same job but the name is confusing. At least it is confusing for my training course attendees. Also the are unfortunately tries to use `asyncio.wait()` because the name is very attractive. But `wait()` doesn't signal about raised exceptions, you know. That's why proper `wait()` usage is: while tasks: done, tasks = await asyncio.wait(tasks, timeout=12) for t in done: await t but nobody call it properly. `asyncio.gather()` is much better alternative, we should promote it. On Mon, Nov 7, 2016 at 9:08 PM Yury Selivanov wrote: > [..] > > Sorry, this was a bit tongue in cheek. This was something I said to > Guido at the *very* beginning of Tulip development, when asked about > mistakes Twisted has made: "don't have a global event loop, you'll never > get away from it". > > > > I still think getting rid of a global loop would always be an > improvement, although I suspect it's too late at this point. `await > current_event_loop()` might make more sense in Asyncio as that's not really > "global", similar to Curio's trap of the same design; however, I assume > that this was an intentional design disagreement for a reason and I don't > see that reason as having changed (as Yury indicates). > > The latest update of get_event_loop is a step in the right direction. At > least now we can document the best practices: > > 1. Have one ?main? coroutine to bootstrap/run your program; > > 2. Don?t design APIs that accept the loop parameter; instead design > coroutine-first APIs and use get_event_loop in your library if you > absolutely need the loop. > > 3. I want to add ?asyncio.main(coro)? function, which would create the > loop, run the ?coro? coroutine, and correctly clean everything up. > > What you propose, IIUC is a step further: > > * Deprecate get_event_loop(); > > * Add ?current_event_loop()? coroutine. > > This will enforce (1) and (2), making asyncio library devs/users to focus > more on coroutines and async/await. > > Am I understanding this all correctly? > > Yury > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Nov 7 14:54:36 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 7 Nov 2016 11:54:36 -0800 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: Message-ID: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> > On Nov 5, 2016, at 5:09 PM, Nathaniel Smith wrote: > > I just posted a long blog/essay that's probably of interest to folks here: > > https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/ > > The short version: I think curio something important to teach us; I > tried to figure out what that is and how we can learn from it. I still haven't had time to read the whole thing yet (there's quite a lot to unpack here!) but I think that might be of interest in examining ways to deal with backpressure that are more declarative; flows are set up ahead of time and then manipulated explicitly as flows, rather than relying on the imperative structure of pseudo-blocking in coroutines. I should note that while Tubes's present implementation is Twisted-specific, the Twisted-specific bits are all around the edges of the system. The core has been explicitly factored to be usable on any event-driven architecture, as long as you have a notion of backpressure and a way to ingest and send data. -glyph From guido at python.org Mon Nov 7 14:58:27 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Nov 2016 11:58:27 -0800 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: <07F33DBD-0EFB-458E-BD69-8476DD4886E3@twistedmatrix.com> References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> <07F33DBD-0EFB-458E-BD69-8476DD4886E3@twistedmatrix.com> Message-ID: I would caution against rushing into anything rash here. Nathaniel's post will stand as one of the most influential posts (about async I/O in Python) of this generation, and curio is a beacon of clarity compared to asyncio. However, asyncio has a much bigger responsibility at this point, as it's in the stdlib, and it must continue to support its existing APIs, on all supported platforms, whether we like them or not. I would love to see a design for a new API that focuses more on coroutines. But it should be a new PEP aimed at Python 3.7 or 3.8. I am tempted to start defending asyncio, but I'll resist, because nothing good can come from that. On Mon, Nov 7, 2016 at 11:41 AM, Glyph Lefkowitz wrote: > > On Nov 7, 2016, at 11:08 AM, Yury Selivanov wrote: > > [..] > > Sorry, this was a bit tongue in cheek. This was something I said to Guido > at the *very* beginning of Tulip development, when asked about mistakes > Twisted has made: "don't have a global event loop, you'll never get away > from it". > > I still think getting rid of a global loop would always be an improvement, > although I suspect it's too late at this point. `await > current_event_loop()` might make more sense in Asyncio as that's not really > "global", similar to Curio's trap of the same design; however, I assume > that this was an intentional design disagreement for a reason and I don't > see that reason as having changed (as Yury indicates). > > > The latest update of get_event_loop is a step in the right direction. At > least now we can document the best practices: > > 1. Have one ?main? coroutine to bootstrap/run your program; > > 2. Don?t design APIs that accept the loop parameter; instead design > coroutine-first APIs and use get_event_loop in your library if you > absolutely need the loop. > > 3. I want to add ?asyncio.main(coro)? function, which would create the > loop, run the ?coro? coroutine, and correctly clean everything up. > > What you propose, IIUC is a step further: > > * Deprecate get_event_loop(); > > * Add ?current_event_loop()? coroutine. > > This will enforce (1) and (2), making asyncio library devs/users to focus > more on coroutines and async/await. > > Am I understanding this all correctly? > > > Yep. It's not so much making users focus *more* on coroutines, as having > a way to pass a loop to a coroutine that is explicit (the coro needs to be > scheduled on a loop already, so the binding has been explicitly specified) > but unobtrusive. > > -glyph > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Mon Nov 7 15:32:27 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Mon, 07 Nov 2016 20:32:27 +0000 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> <07F33DBD-0EFB-458E-BD69-8476DD4886E3@twistedmatrix.com> Message-ID: Guido, from my perspective asyncio is just great. Because: 1. It raised striving for better coroutine based solutions like curio. We could borrow best ideas and incorporate them into asyncio. I don't want to rush, don't get me wrong please. Maybe some third-party library built on top of asyncio but designed with curio spirit for proving the concept. BTW Curio was born after async/await syntax which was in turn born by real asyncio use cases. 2. asyncio was designed as the grand basement standard for asynchronous development. It *right now* is supported by Tornado and Twisted. The best practice for tornado users (if they has been ported their codebase to Python 3) is using asyncio compatible libraries for communicating with databases etc AFAIC. I'm considering this as very big win for both asyncio and tornado/twisted. But this way is possible only because asyncio supports futures, callbacks, protocols as low-level API. On Mon, Nov 7, 2016 at 9:58 PM Guido van Rossum wrote: > I would caution against rushing into anything rash here. Nathaniel's post > will stand as one of the most influential posts (about async I/O in Python) > of this generation, and curio is a beacon of clarity compared to asyncio. > However, asyncio has a much bigger responsibility at this point, as it's in > the stdlib, and it must continue to support its existing APIs, on all > supported platforms, whether we like them or not. > > I would love to see a design for a new API that focuses more on > coroutines. But it should be a new PEP aimed at Python 3.7 or 3.8. > > I am tempted to start defending asyncio, but I'll resist, because nothing > good can come from that. > > On Mon, Nov 7, 2016 at 11:41 AM, Glyph Lefkowitz > wrote: > > > On Nov 7, 2016, at 11:08 AM, Yury Selivanov wrote: > > [..] > > Sorry, this was a bit tongue in cheek. This was something I said to Guido > at the *very* beginning of Tulip development, when asked about mistakes > Twisted has made: "don't have a global event loop, you'll never get away > from it". > > I still think getting rid of a global loop would always be an improvement, > although I suspect it's too late at this point. `await > current_event_loop()` might make more sense in Asyncio as that's not really > "global", similar to Curio's trap of the same design; however, I assume > that this was an intentional design disagreement for a reason and I don't > see that reason as having changed (as Yury indicates). > > > The latest update of get_event_loop is a step in the right direction. At > least now we can document the best practices: > > 1. Have one ?main? coroutine to bootstrap/run your program; > > 2. Don?t design APIs that accept the loop parameter; instead design > coroutine-first APIs and use get_event_loop in your library if you > absolutely need the loop. > > 3. I want to add ?asyncio.main(coro)? function, which would create the > loop, run the ?coro? coroutine, and correctly clean everything up. > > What you propose, IIUC is a step further: > > * Deprecate get_event_loop(); > > * Add ?current_event_loop()? coroutine. > > This will enforce (1) and (2), making asyncio library devs/users to focus > more on coroutines and async/await. > > Am I understanding this all correctly? > > > Yep. It's not so much making users focus *more* on coroutines, as having > a way to pass a loop to a coroutine that is explicit (the coro needs to be > scheduled on a loop already, so the binding has been explicitly specified) > but unobtrusive. > > -glyph > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Nov 7 19:18:19 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 7 Nov 2016 16:18:19 -0800 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> <07F33DBD-0EFB-458E-BD69-8476DD4886E3@twistedmatrix.com> Message-ID: > On Nov 7, 2016, at 11:58 AM, Guido van Rossum wrote: > > I would caution against rushing into anything rash here. Nathaniel's post will stand as one of the most influential posts (about async I/O in Python) of this generation, and curio is a beacon of clarity compared to asyncio. However, asyncio has a much bigger responsibility at this point, as it's in the stdlib, and it must continue to support its existing APIs, on all supported platforms, whether we like them or not. My smiley may have been insufficiently forceful. I was not intending to seriously suggest a departure from the current API. A 3.7/3.8 refinement into preferring a 'current_event_loop' coroutine might be a nice future direction but it is not something that should happen lightly. > I would love to see a design for a new API that focuses more on coroutines. But it should be a new PEP aimed at Python 3.7 or 3.8. > > I am tempted to start defending asyncio, but I'll resist, because nothing good can come from that. TBH I think that this discussion stems from a strength of asyncio's design, not a weakness. As David did, let me underscore Brett's comment: the fact that asyncio has multiple, separable layers which each interact via well-defined interfaces has allowed for a tremendous amount of experimentation and refinement. Most languages with async features are locked into a particular substrate, and languages without async features end up being an uncoordinated mess of incompatible APIs. I feel like we're really getting the best of both worlds: language-level support with interoperability and ecosystem considerations baked in right from the start. The potential for growth and improvement necessarily comes along with disagreement and criticism but it seems like overall this is a very healthy development. Right now we're talking about this at the async event layer, but previous work at the loop layer (uvloop) also points in exciting future directions for community improvements that maintain interoperability across the whole ecosystem. All the integration points exposed in asyncio's design already seem to be benefiting from community-wide scrutiny and tinkering. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 7 19:33:51 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Nov 2016 16:33:51 -0800 Subject: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world In-Reply-To: References: <6D67AECE-5B10-4D60-9D4E-5E0D11D63ECC@twistedmatrix.com> <72A616B2-68AF-43DE-965B-4F0B32EF83AA@gmail.com> <8B59FAE7-7F0C-420E-8421-362B3D213314@twistedmatrix.com> <3898952D-3517-4074-948D-EEB09D126B91@gmail.com> <07F33DBD-0EFB-458E-BD69-8476DD4886E3@twistedmatrix.com> Message-ID: On Mon, Nov 7, 2016 at 4:18 PM, Glyph Lefkowitz wrote: > > On Nov 7, 2016, at 11:58 AM, Guido van Rossum wrote: > > I would caution against rushing into anything rash here. Nathaniel's post > will stand as one of the most influential posts (about async I/O in Python) > of this generation, and curio is a beacon of clarity compared to asyncio. > However, asyncio has a much bigger responsibility at this point, as it's in > the stdlib, and it must continue to support its existing APIs, on all > supported platforms, whether we like them or not. > > > My smiley may have been insufficiently forceful. I was not intending to > seriously suggest a departure from the current API. A 3.7/3.8 refinement > into preferring a 'current_event_loop' coroutine might be a nice future > direction but it is not something that should happen lightly. > It wasn't aimed at you, it was aimed at asyncio's staunchest supporter/maintainers. (Including myself. :-) > > I would love to see a design for a new API that focuses more on > coroutines. But it should be a new PEP aimed at Python 3.7 or 3.8. > > I am tempted to start defending asyncio, but I'll resist, because nothing > good can come from that. > > > TBH I think that this discussion stems from a *strength* of asyncio's > design, not a weakness. As David did, let me underscore Brett's comment: > the fact that asyncio has multiple, separable layers which each interact > via well-defined interfaces has allowed for a tremendous amount of > experimentation and refinement. Most languages with async features are > locked into a particular substrate, and languages without async features > end up being an uncoordinated mess of incompatible APIs. I feel like we're > really getting the best of both worlds: language-level support with > interoperability and ecosystem considerations baked in right from the start. > > The potential for growth and improvement necessarily comes along with > disagreement and criticism but it seems like overall this is a very healthy > development. > > Right now we're talking about this at the async event layer, but previous > work at the loop layer (uvloop) also points in exciting future directions > for community improvements that maintain interoperability across the whole > ecosystem. All the integration points exposed in asyncio's design already > seem to be benefiting from community-wide scrutiny and tinkering. > I think Nathaniel expressed his appreciation for this as well (especially in his acknowledgments section). In terns of Python's historic development, I think it's fascinating that this can still be seen as a development building on Python's approach to iteration: In Python 0.0 all we had was `for x in ` (I believe even then, it was possible to write extension types in C that also supported iteration, but I'm not sure). We then expanded upon this to add `__iter__` and `__next__` (n?? `next`), then yield, then send() and throw(), then `yield from` (I still regret we didn't manage to get this into 2.7 due to some stupid process issue), then the asyncio library, then async/await, and it seems this (degenerate :-) tree will still bear more fruit in the future. (And I may even have missed a PEP or two.) Just imagine the alternate universe where Python 0.0 had borrowed its for-loop from C instead of from ABC! Which means the seed of all this goes back over 35 years. I hope Lambert Meertens and Leo Geurts are proud. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Tue Nov 8 05:17:08 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 8 Nov 2016 10:17:08 +0000 Subject: [Async-sig] APIs for backpressure (was: Re: Some thoughts on asynchronous API design in a post-async/await world) In-Reply-To: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> References: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> Message-ID: > On 7 Nov 2016, at 19:54, Glyph Lefkowitz wrote: > > I still haven't had time to read the whole thing yet (there's quite a lot to unpack here!) but I think that might be of interest in examining ways to deal with backpressure that are more declarative; flows are set up ahead of time and then manipulated explicitly as flows, rather than relying on the imperative structure of pseudo-blocking in coroutines. I?d like to fork the discussion off to talk about backpressure for a while. I think the issue of propagating backpressure is one that is really well suited to this list, because even *knowing* about backpressure is the marker of an engineer who has designed complex asynchronous systems. I suspect this list has a higher proportion of people capable of talking sensibly about backpressure propagation than most. For my part, I actually think the backpressure discussion in Nathaniel?s post was the most interesting *mechanical* part of the post. Nathaniel has correctly identified that any system that does buffered writes/reads (e.g. asyncio and Twisted) needs powerful systems for propagating backpressure in a sensible way. If we want people to really develop resilient asynchronous systems in Python it must be possible for those developers to be able to sensibly propagate backpressure through the system, and ideally to fall into a pit of success whereby the things developers naturally do propagate backpressure as a matter of course. Unfortunately I?d like to suggest that neither Twisted nor asyncio are in possession of really great APIs for exerting and managing backpressure. Twisted?s IPushProducer/IConsumer interface is present and moderately effective, but it has a few limits. The most frustrating limitation of this design is that it does not allow for easy construction of a pipeline: an object can implement IConsumer/IPushProducer in only one ?direction?: that is, if you have a chain of objects A <-> B <-> C, B can propagate backpressure only to C or to A. That?s problematic for protocols like HTTP/2 which require balancing backpressure in multiple directions at once. An additional problem is that Twisted?s APIs here are one-to-one: that is, each B can only have one A and one C associated with it. That makes it very hard to do a fan-in/fan-out design with IPushProducer or IConsumer. Both of these problems can be worked around of course: the creation of proxy objects or implicit interfaces between multiple objects can allow this to work, and that?s effectively what Twisted?s HTTP/2 layer does. But these are complex APIs, and they are definitely expert-oriented. On top of this, the unfriendliness of that interface means that developers are likely to consider it an optional part of their protocol implementation and will thus fail to exert backpressure at all. (All of the problems of Twisted?s interface here apply to asyncio, by the by, I just happen to be able to talk in more detail about Twisted?s.) Tubes is unquestionably a better API for this, but suffers from a lack of accessibility. It would definitely be interesting to see if tubes can be easily replicated on top of asyncio (and indeed curio), because at least for me something like tubes is what I?ve wanted for a long time in the Python world. If the design of tubes is interesting to the asyncio team, it would justify spending more time trying to integrate it While we?re on the topic, we should also discuss forms of backpressure response that even curio doesn?t neatly handle. For example, in many systems that would like to be highly responsive it is common to want to propagate backpressure to the edge of the system and then to use that information to rapidly provide error messages to inbound work items (in HTTP speak, if your WSGI server is overloaded it would often be better to rapidly provide a 503 Service Unavailable response with a Retry-After header than to sit on the request for a potentially unbounded amount of time. The curio backpressure propagation mechanisms that Nathaniel outlined do not tolerate this situation well at all (not a slight on curio, just a natural extension of the behaviour of the socket-like APIs). I?m interested to see what other thoughts people in this space have though. Do alternative API designs seem sensible to people? Cory From multisosnooley at gmail.com Mon Nov 7 18:18:36 2016 From: multisosnooley at gmail.com (MultiSosnooley .) Date: Tue, 8 Nov 2016 02:18:36 +0300 Subject: [Async-sig] sans io implementation In-Reply-To: References: Message-ID: Alex, I don't get why generators are bad for sans io? For me It's just simple state holder. Some things need multiply steps for making one high-level action. And in this case you can make class, which will hold state between calls and switch with "ugly" if/elif/elif/.../elif/else or you can use generator, so protocol code will have readable flow and looks like it is with io. Probably I missed something about generators? On Mon, Oct 24, 2016 at 8:08 PM, MultiSosnooley . wrote: > Hi. > > I'm trying to make sans-io library. > > 1. Protocol works over http, so I've just passing Request object with > method, url, data, etc. fields to user and receive Response object (json in > this case). > 2. I'm using generators for simplification of holding state of > multirequest actions. Something like this: > > def login(): > > response = yield Request(...) # check server state > # update state with response > response = yield Request(...) # actually login > yield None > > def act(generator): > response = None > while True: > request = generator.send(response) > if request is None: > return response > response = # do io > > act(login()) > > This solve problem multistep actions. The downside is that all your > protocol functions, event if they have only one request and don't need > response at all, must be generators. > > Is it ok to send just abstract Request and receive abstract Response (not > just bytes)? > Is there a better solution for multirequest actions? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Nov 8 16:07:30 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 8 Nov 2016 13:07:30 -0800 Subject: [Async-sig] APIs for backpressure (was: Re: Some thoughts on asynchronous API design in a post-async/await world) In-Reply-To: References: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> Message-ID: > On Nov 8, 2016, at 2:17 AM, Cory Benfield wrote: > > Tubes is unquestionably a better API for this, but suffers from a lack of accessibility. What is "accessibility" in this context? > It would definitely be interesting to see if tubes can be easily replicated on top of asyncio (and indeed curio), because at least for me something like tubes is what I?ve wanted for a long time in the Python world. If the design of tubes is interesting to the asyncio team, it would justify spending more time trying to integrate it I obviously can't speak to the interest of the asyncio team, but binding it to asyncio (you shouldn't need to "replicate" it, the library is designed to be portable to multiple backends) should be pretty easy. All you need to do is to write asyncio versions of these two modules: https://github.com/twisted/tubes/blob/2089781479a8f4a2d3027c88560bb5f39cfd90a1/tubes/protocol.py https://github.com/twisted/tubes/blob/2089781479a8f4a2d3027c88560bb5f39cfd90a1/tubes/undefer.py That's <500 lines of heavily documented, generously spaced code. Nothing else in the package ought to import Twisted. Given the similarity of asyncio's and Twisted's low-level callback interfaces, I imagine the translation would be fairly literal; the only real stumbling block is the lack of interfaces like IStreamServerEndpoint and IStreamClientEndpoint in asyncio. However, given that these interfaces are just one method each, adding a literal equivalent that just calls event_loop.create_connection and event_loop.create_server just for Tubes would give you _exactly_ the same interface. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Nov 8 16:21:04 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Tue, 8 Nov 2016 13:21:04 -0800 Subject: [Async-sig] APIs for backpressure (was: Re: Some thoughts on asynchronous API design in a post-async/await world) In-Reply-To: References: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> Message-ID: <9B2B9D42-10ED-4F5E-90AA-4FAC8B37B38A@twistedmatrix.com> > On Nov 8, 2016, at 1:07 PM, Glyph Lefkowitz wrote: > > That's <500 lines of heavily documented, generously spaced code. Nothing else in the package ought to import Twisted. Oh, um. There's some protocol parsing code that incidentally depends on Twisted right now too, but that dependency could easily be eliminated (and it doesn't touch any reactor APIs, so you can ignore it for the purposes of porting as long as Twisted is installed). -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Wed Nov 9 10:59:06 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 9 Nov 2016 15:59:06 +0000 Subject: [Async-sig] APIs for backpressure (was: Re: Some thoughts on asynchronous API design in a post-async/await world) In-Reply-To: References: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> Message-ID: > On 8 Nov 2016, at 21:07, Glyph Lefkowitz wrote: > > >> On Nov 8, 2016, at 2:17 AM, Cory Benfield > wrote: >> >> Tubes is unquestionably a better API for this, but suffers from a lack of accessibility. > > What is "accessibility" in this context? ?accessibility? in this context is essentially the collection of things that make it easy for users to a) identify a need for tubes, b) work out how to plug tubes into their application, and c) have a sensible evolution to handle evolving backpressure needs into tubes. Mostly this is a documentation thing, but there?s also a chicken-and-egg problem here, specifically: tubes provides a high-level API for flow control but requires that pre-existing code use a low-level one. How do we get from there to somewhere we can actually tell people ?yeah, go use tubes?? On top of that we have: how do we justify using tubes when so much of for example Twisted?s codebase does not implement IPushProducer/IConsumer? How do people migrate a pre-existing codebase to something like tubes? How do people extend tubes to do something other than *propagate* backpressure (e.g. to implement a fast-fail path to error out rather than stop reading from a socket). All of these questions *have* answers, but those answers aren?t easily accessible. Part of this is an ongoing cultural problem which is that people who build small or non-distributed applications often don?t have to think about backpressure, so there?s another problem that also needs addressing: it needs to be so easy for people to extend their async producers and consumers of data to propagate and respond to backpressure appropriately that there?s no good reason *not* to do it. All of this complex mess of things is what I mean by ?accessibility?. It needs to be easier to do the right thing than the wrong thing. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Wed Nov 9 21:43:46 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Wed, 9 Nov 2016 18:43:46 -0800 Subject: [Async-sig] APIs for backpressure (was: Re: Some thoughts on asynchronous API design in a post-async/await world) In-Reply-To: References: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> Message-ID: <4B3ABF7E-1039-4FE5-9E07-163251490FCB@twistedmatrix.com> > On Nov 9, 2016, at 7:59 AM, Cory Benfield wrote: > > >> On 8 Nov 2016, at 21:07, Glyph Lefkowitz > wrote: >> >> >>> On Nov 8, 2016, at 2:17 AM, Cory Benfield > wrote: >>> >>> Tubes is unquestionably a better API for this, but suffers from a lack of accessibility. >> >> What is "accessibility" in this context? > > ?accessibility? in this context is essentially the collection of things that make it easy for users to a) identify a need for tubes, b) work out how to plug tubes into their application, and c) have a sensible evolution to handle evolving backpressure needs into tubes. I see what you mean. I've struggled with this problem a lot myself. > Mostly this is a documentation thing, but there?s also a chicken-and-egg problem here, specifically: tubes provides a high-level API for flow control but requires that pre-existing code use a low-level one. How do we get from there to somewhere we can actually tell people ?yeah, go use tubes?? Aah, yes. This is exactly where the project is stuck. A large amount of infrastructure has to be retrofitted to be Tubes all the way down before you can make them useful. And the problem isn't just with Tubes at the infrastructure level; most applications are fundamentally not stream processing, and it will be idiomatically challenging to express them as such. HTTP connections are short-lived and interfaces to comprehending them (i.e. 'json.loads') are not themselves stream oriented. Even if you did have a stream-oriented JSON parser, expressing what you want from it is hard; you want a data structure that you can simultaneously inspect multiple elements from, not a stream of "object began" / "string began" / "string ended" / "list began" / "list ended" events. > On top of that we have: how do we justify using tubes when so much of for example Twisted?s codebase does not implement IPushProducer/IConsumer? I think this may be slightly misleading. You're making it sound like there is a proliferation of transports or stream interfaces that don't provide these interfaces, but should. At a low level, almost everything in the Twisted codebase which is actually a _stream_ of data (rather than a request/response) does implement these interfaces, or has a public `transport` attribute which does so. For example, the HTTP response object that comes back from Agent does have a transport. The issue is not that the interfaces aren't provided on specific objects where they should be, but that the entire shape of the arbitrarily-large-request/arbitrarily-large-response pattern expects to be able to store whole responses. By the time you get to the layer which "doesn't implement" IPushProducer/IConsumer, you're at a level where such an implementation would be meaningless, or unhelpful. > How do people migrate a pre-existing codebase to something like tubes? How do people extend tubes to do something other than *propagate* backpressure (e.g. to implement a fast-fail path to error out rather than stop reading from a socket). All of these questions *have* answers, but those answers aren?t easily accessible. I think one way to start to drill into this would be (sorry for the over-specificty to Twisted here) to address something like with Tubes. There are at least a few clear-cut cases where we _do_ have a large stream of data which needs to be directed to an appropriate location, and the sooner we can make the standard interface for that into "return a Fount", the sooner we can start to make Tubes just as much of the idiomatic lexicon of async I/O as awaitables, Futures or Deferreds. Pulling a dependency like this into asyncio would obviously be challenging, but 3rd-party packages like Twisted - or, for that matter, aiohttp! - could start to depend on tubes as-is. > Part of this is an ongoing cultural problem which is that people who build small or non-distributed applications often don?t have to think about backpressure, so there?s another problem that also needs addressing: it needs to be so easy for people to extend their async producers and consumers of data to propagate and respond to backpressure appropriately that there?s no good reason *not* to do it. The extension of this cultural problem is that most of the tools used in large distributed systems are built as hobby projects for small, non-distributed systems, so even at scale and in these environments we still find ourselves fighting with layers that don't want to deal with backpressure properly. More importantly, backpressure at scale in distributed systems often means really weird stuff, like, traffic shaping on a front-end tier by coordinating with a data store or back-end tier to identify problem networks or network ranges. Tubes operates at a simpler level: connections are individual entities, and backpressure is applied uniformly across all of them. Granted, this is the basic layer you need in place to make addressing backpressure throughout a system work properly, but it's also not an exciting product that solves a super hard or complex problem. > All of this complex mess of things is what I mean by ?accessibility?. It needs to be easier to do the right thing than the wrong thing. I'm definitely open to more ideas on this topic. Retrofitting backpressure into existing systems is hard, and harder still when you're trying to expose an idiomatic, high-level API. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Nov 10 06:10:52 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 10 Nov 2016 11:10:52 +0000 Subject: [Async-sig] APIs for backpressure (was: Re: Some thoughts on asynchronous API design in a post-async/await world) In-Reply-To: <4B3ABF7E-1039-4FE5-9E07-163251490FCB@twistedmatrix.com> References: <3707CDC3-311D-426C-9C7E-E48A9DD2A57A@twistedmatrix.com> <4B3ABF7E-1039-4FE5-9E07-163251490FCB@twistedmatrix.com> Message-ID: <4D77168B-9CFF-4B02-BD18-D61DB0EF4D03@lukasa.co.uk> > On 10 Nov 2016, at 02:43, Glyph Lefkowitz wrote: > > And the problem isn't just with Tubes at the infrastructure level; most applications are fundamentally not stream processing, and it will be idiomatically challenging to express them as such. HTTP connections are short-lived and interfaces to comprehending them (i.e. 'json.loads') are not themselves stream oriented. Even if you did have a stream-oriented JSON parser, expressing what you want from it is hard; you want a data structure that you can simultaneously inspect multiple elements from, not a stream of "object began" / "string began" / "string ended" / "list began" / "list ended" events. [snip] > > More importantly, backpressure at scale in distributed systems often means really weird stuff, like, traffic shaping on a front-end tier by coordinating with a data store or back-end tier to identify problem networks or network ranges. Tubes operates at a simpler level: connections are individual entities, and backpressure is applied uniformly across all of them. Granted, this is the basic layer you need in place to make addressing backpressure throughout a system work properly, but it's also not an exciting product that solves a super hard or complex problem. So these two things here are the bit I?m most interested in focusing on. You?re totally right: backpressure in stream oriented systems is most effectively managed in the form that tubes allows. However, many systems are not stream-oriented but instead focus on quanta of work. However, backpressure is still a real part of system design for systems like that, and it would be good to have a higher-level API for designing backpressure propagation for quantised work. The biggest problem there I think is that there?s not just one way to do that. For example, a common mechanism is to provide something like a token bucket at the edges of your system whereby you allow for only N work items to be outstanding at any one time. The obvious problem with this is that there is no one true value for N: it depends on how intensive your work items are on the system and what their latency/throughput characteristics look like. That means that, for example, Twisted cannot simply choose a value for this for its users. At this point we?re probably off into the weeds though. More important, I think, is to make a system that is amenable to having something like a token bucket attached to it and integrated into the stream-based flow control mechanisms. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Nov 15 01:52:23 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 14 Nov 2016 22:52:23 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released Message-ID: Hi all, I just released v1.2 of my async_generator package: https://pypi.org/pypi/async_generator/ This package makes it easy to write PEP 525-style "async generators", even on Python 3.5. It looks like: from async_generator import async_generator, yield_, yield_from_ @async_generator async def parse_json_separated_newlines_from_network(streamreader): async for line in streamreader: await yield_(json.loads(line)) Starting in v1.1 (released last week but not announced widely), the generators are now highly compatible with 3.6's native async generators, including full support for athrow / asend / aclose. They also fully support "yield from", which is otherwise unavailable even on Python 3.6. v1.2 fixed some edge cases in "yield from" -- but the headline feature is that we're now *so* compatible with 3.6's native async generators that you can now use our yield from support directly inside a native generator: from async_generator import yield_from_ async def f(): yield 2 yield 3 # yields [1, 2, 3, 4] async def g(): yield 1 await yield_from_(f()) yield 4 Have fun, -n -- Nathaniel J. Smith -- https://vorpus.org From yselivanov at gmail.com Wed Nov 16 13:12:42 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 16 Nov 2016 13:12:42 -0500 Subject: [Async-sig] Adding asyncio.run() function in Python 3.6 Message-ID: One of the remaining problems with the event loop in asyncio is bootstrapping/finalizing asyncio programs. Currently, there are two different scenarios: [1] Running a coroutine: async def main(): # your program loop = asyncio.get_event_loop() try: loop.run_until_complete(main()) finally: loop.close() [2] Running a server: loop = asyncio.get_event_loop() srv = loop.run_until_complete( loop.create_server(?)) try: loop.run_forever() finally: try: srv.close() loop.run_until_complete(srv.wait_closed()) finally: loop.close() Both cases are *incomplete*: they don?t do correct finalization of asynchronous generators. To do that we?ll need to add another 1-3 lines of code (extra try-finally). This manual approach is really painful: * It makes bootstrapping asyncio code unnecessarily hard. * It makes the documentation hard to follow. And we can?t restructure the docs to cover the loop only in the advanced section. * Most of the people will never know about `loop.shutdown_asyncgen` coroutine. * We don?t have a place to add debug code to let people know that their asyncio program didn?t clean all resources properly (a lot of unordered warnings will be spit out instead). In https://github.com/python/asyncio/pull/465 I propose to add a new function to asyncio in Python 3.6: asyncio.run(). The function can either accept a coroutine, which solves [1]: async def main(): # your program asyncio.run(main()) Or it can accept an asynchronous generator, which solves [2]: async def main(): srv = await loop.create_server(?)) try: yield # let the loop run forever finally: srv.close() await srv.wait_closed() asyncio.run(main()) asyncio.run() solves the following: * An easy way to start an asyncio program that properly takes care of loop instantiation and finalization. * It looks much better in the docs. With asyncio.run people don?t need to care about the loop at all, most probably will never use it. * Easier to experiment with asyncio in REPL. * The loop and asynchronous generators will be cleaned up properly. * We can add robust debug output to the function, listing the unclosed tasks, servers, connections, asynchronous generators etc, helping people with the cleanup logic. * Later, if we need to add more cleanup code to asyncio, we will have a function to add the logic to. I feel that we should add this to asyncio. One of the arguments against that, is that overloading asyncio.run to accept both coroutines and asynchronous generators makes the API more complex. If that?s really the case, we can add two functions: asyncio.run(coro) and asyncio.run_forever(async_generator). Also take a look at https://github.com/python/asyncio/pull/465. Thanks, Yury From njs at pobox.com Wed Nov 16 15:35:08 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 16 Nov 2016 12:35:08 -0800 Subject: [Async-sig] Adding asyncio.run() function in Python 3.6 In-Reply-To: References: Message-ID: What's the use case for the async generator version? Could the yield be replaced by 'await loop.shutting_down()'? On Nov 16, 2016 10:12 AM, "Yury Selivanov" wrote: > One of the remaining problems with the event loop in asyncio is > bootstrapping/finalizing asyncio programs. > > Currently, there are two different scenarios: > > [1] Running a coroutine: > > async def main(): > # your program > loop = asyncio.get_event_loop() > try: > loop.run_until_complete(main()) > finally: > loop.close() > > [2] Running a server: > > loop = asyncio.get_event_loop() > srv = loop.run_until_complete( > loop.create_server(?)) > try: > loop.run_forever() > finally: > try: > srv.close() > loop.run_until_complete(srv.wait_closed()) > finally: > loop.close() > > Both cases are *incomplete*: they don?t do correct finalization of > asynchronous generators. To do that we?ll need to add another 1-3 lines of > code (extra try-finally). > > This manual approach is really painful: > > * It makes bootstrapping asyncio code unnecessarily hard. > > * It makes the documentation hard to follow. And we can?t restructure the > docs to cover the loop only in the advanced section. > > * Most of the people will never know about `loop.shutdown_asyncgen` > coroutine. > > * We don?t have a place to add debug code to let people know that their > asyncio program didn?t clean all resources properly (a lot of unordered > warnings will be spit out instead). > > In https://github.com/python/asyncio/pull/465 I propose to add a new > function to asyncio in Python 3.6: asyncio.run(). > > The function can either accept a coroutine, which solves [1]: > > async def main(): > # your program > asyncio.run(main()) > > Or it can accept an asynchronous generator, which solves [2]: > > async def main(): > srv = await loop.create_server(?)) > try: > yield # let the loop run forever > finally: > srv.close() > await srv.wait_closed() > > asyncio.run(main()) > > asyncio.run() solves the following: > > * An easy way to start an asyncio program that properly takes care of loop > instantiation and finalization. > > * It looks much better in the docs. With asyncio.run people don?t need to > care about the loop at all, most probably will never use it. > > * Easier to experiment with asyncio in REPL. > > * The loop and asynchronous generators will be cleaned up properly. > > * We can add robust debug output to the function, listing the unclosed > tasks, servers, connections, asynchronous generators etc, helping people > with the cleanup logic. > > * Later, if we need to add more cleanup code to asyncio, we will have a > function to add the logic to. > > I feel that we should add this to asyncio. One of the arguments against > that, is that overloading asyncio.run to accept both coroutines and > asynchronous generators makes the API more complex. If that?s really the > case, we can add two functions: asyncio.run(coro) and > asyncio.run_forever(async_generator). > > Also take a look at https://github.com/python/asyncio/pull/465. > > Thanks, > Yury > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Wed Nov 16 15:53:50 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Wed, 16 Nov 2016 15:53:50 -0500 Subject: [Async-sig] Adding asyncio.run() function in Python 3.6 In-Reply-To: References: Message-ID: <1A70D245-C1CE-4026-B4B8-9BC9FF74A9B8@gmail.com> > On Nov 16, 2016, at 3:35 PM, Nathaniel Smith wrote: > > What's the use case for the async generator version? Could the yield be replaced by 'await loop.shutting_down()?? Async generator version (inspired by contextlib.contextmanager decorator) is needed in cases where you want loop.run_forever. The PR originally proposed to add `asyncio.forever()` (which is the same idea as `loop.shutting_down()`), but nobody particularly liked it. A couple of thoughts/reasons: 1. Some pretty intrusive modifications are required to be made in the event loop to make it work. That means all other event loops (including uvloop) will have to be modified to support it. This is the most important reason. 2. `loop.shutting_down()` is a no go because it?s a method on the loop object. We can discuss `asyncio.shutting_down`. The whole point of this discussion is to get rid of the event loop. 3. `await forever()` and `await shutting_down()` have a naming issue - both look weird: async def main(): srv = await asyncio.start_server(?) try: await asyncio.shutting_down() # or await forever() finally: srv.close() await srv.wait_closed() In the above example, what does the second ?await? do? Will it be resolved when the loop is stopped with ?loop.stop()?? Or when a KeyboardInterrupt occurs? What will happen if you await on it in parallel from 10 different coroutines? It?s just difficult to define a clear semantics of this coroutine. Using an asynchronous generator for this case is easier in terms of implementation and in terms of specifying the execution semantics. And the approach of using generators for such things isn?t new - we have contextlib.contextmanager decorator which is quite popular. Yury From rwilliams at lyft.com Thu Nov 17 13:17:33 2016 From: rwilliams at lyft.com (Roy Williams) Date: Thu, 17 Nov 2016 10:17:33 -0800 Subject: [Async-sig] Adding asyncio.run() function in Python 3.6 In-Reply-To: <1A70D245-C1CE-4026-B4B8-9BC9FF74A9B8@gmail.com> References: <1A70D245-C1CE-4026-B4B8-9BC9FF74A9B8@gmail.com> Message-ID: Yury this looks great! Thanks for suggesting this. One of the big problems I've seen with the basic approachability of async/await in Python is the lack of interpreter support. Now this makes that a lot easier by having a one function you can use makes this a lot easier (One of the main things I use `curio` for is simply being able to `curio.run` an Awaitable in the interpreter without having to go through the normal asyncio dance). I would love to see if we could extend this even further to making `await` "Just Work" in the interpreter. Thanks, Roy On Wed, Nov 16, 2016 at 12:53 PM, Yury Selivanov wrote: > > > On Nov 16, 2016, at 3:35 PM, Nathaniel Smith wrote: > > > > What's the use case for the async generator version? Could the yield be > replaced by 'await loop.shutting_down()?? > > Async generator version (inspired by contextlib.contextmanager decorator) > is needed in cases where you want loop.run_forever. > > The PR originally proposed to add `asyncio.forever()` (which is the same > idea as `loop.shutting_down()`), but nobody particularly liked it. > > A couple of thoughts/reasons: > > 1. Some pretty intrusive modifications are required to be made in the > event loop to make it work. That means all other event loops (including > uvloop) will have to be modified to support it. This is the most important > reason. > > 2. `loop.shutting_down()` is a no go because it?s a method on the loop > object. We can discuss `asyncio.shutting_down`. The whole point of this > discussion is to get rid of the event loop. > > 3. `await forever()` and `await shutting_down()` have a naming issue - > both look weird: > > async def main(): > srv = await asyncio.start_server(?) > try: > await asyncio.shutting_down() # or await forever() > finally: > srv.close() > await srv.wait_closed() > > In the above example, what does the second ?await? do? Will it be > resolved when the loop is stopped with ?loop.stop()?? Or when a > KeyboardInterrupt occurs? What will happen if you await on it in parallel > from 10 different coroutines? It?s just difficult to define a clear > semantics of this coroutine. > > Using an asynchronous generator for this case is easier in terms of > implementation and in terms of specifying the execution semantics. And the > approach of using generators for such things isn?t new - we have > contextlib.contextmanager decorator which is quite popular. > > Yury > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vxgmichel at gmail.com Fri Nov 18 06:05:29 2016 From: vxgmichel at gmail.com (Vincent Michel) Date: Fri, 18 Nov 2016 12:05:29 +0100 Subject: [Async-sig] Adding asyncio.run() function in Python 3.6 In-Reply-To: References: <1A70D245-C1CE-4026-B4B8-9BC9FF74A9B8@gmail.com> Message-ID: > One of the big problems I've seen with the basic approachability of > async/await in Python is the lack of interpreter support. You might want to have a look at aioconsole [1]; it provides an asynchronous REPL that you can use to interact with asyncio servers. You can try it out using the apython script: $ apython Python 3.5.0 (default, Sep 7 2015, 14:12:03) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. --- This console is running in an asyncio event loop. It allows you to wait for coroutines using the 'await' syntax. Try: await asyncio.sleep(1, result=3) --- >>> await asyncio.sleep(1, result=3) # Wait one second... 3 >>> There is also two ipython extensions: - ipython-yf [2] - asyncio-ipython-magic [3] [1] https://github.com/vxgmichel/aioconsole [2] https://github.com/tecki/ipython-yf [3] https://github.com/Gr1N/asyncio-ipython-magic Cheers, /Vincent From yselivanov at gmail.com Fri Nov 18 18:02:53 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 18 Nov 2016 18:02:53 -0500 Subject: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future In-Reply-To: References: <408d5393-82e4-4176-8d37-5c3d086c4087@googlegroups.com> Message-ID: > On Nov 18, 2016, at 5:53 PM, Luca Sbardella wrote: > > But tests taking 1.48 longer to run on average! > Anything I should know about 3.6 and performance? > That shouldn?t happen. Are you sure you aren?t running them in debug mode? Try to comment out imports of ?_asyncio? in futures.py and tasks.py and run benchmarks in 3.6 to compare Py Futures to C Futures. Also, which Python 3.6 version are you using? Please try to build one from the repo, I?ve fixed a couple of bugs since 3.6b2. Yury From yselivanov at gmail.com Fri Nov 18 18:09:44 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 18 Nov 2016 18:09:44 -0500 Subject: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future In-Reply-To: References: <408d5393-82e4-4176-8d37-5c3d086c4087@googlegroups.com> Message-ID: <04164175-A2AF-437C-A701-735CC49F33EE@gmail.com> Also, are you using uvloop or vanilla asyncio? Try to benchmark vanilla first. And if you have time, please try to test different combinations on vanilla asyncio: Python 3.5 + vanilla asyncio Python 3.6 + vanilla asyncio Python 3.6 + Py Future + Py Task Python 3.6 + Py Future + C Task Python 3.6 + C Future + C Task Python 3.6 + Py Future + Py Task Yury > On Nov 18, 2016, at 6:02 PM, Yury Selivanov wrote: > > >> On Nov 18, 2016, at 5:53 PM, Luca Sbardella wrote: >> >> But tests taking 1.48 longer to run on average! >> Anything I should know about 3.6 and performance? >> > > > That shouldn?t happen. Are you sure you aren?t running them in debug mode? Try to comment out imports of ?_asyncio? in futures.py and tasks.py and run benchmarks in 3.6 to compare Py Futures to C Futures. > > Also, which Python 3.6 version are you using? Please try to build one from the repo, I?ve fixed a couple of bugs since 3.6b2. > > Yury From yselivanov at gmail.com Fri Nov 18 18:19:10 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Fri, 18 Nov 2016 18:19:10 -0500 Subject: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future In-Reply-To: References: <408d5393-82e4-4176-8d37-5c3d086c4087@googlegroups.com> Message-ID: > On Nov 18, 2016, at 6:16 PM, Luca Sbardella wrote: > > Running 3.6-dev from travis, currently b4+. > No debug mode, that is an order of magnitude slower ;-) FWIW I found that I can?t really trust the build times on travis for measuring any kind of performance regressions. You don?t know how it?s compiled there, and more importantly, if Python 3.6 is compiled with the same settings as 3.5. Yury From luca.sbardella at gmail.com Fri Nov 18 17:42:51 2016 From: luca.sbardella at gmail.com (Luca Sbardella) Date: Fri, 18 Nov 2016 22:42:51 +0000 Subject: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future In-Reply-To: References: <408d5393-82e4-4176-8d37-5c3d086c4087@googlegroups.com> Message-ID: Pulsar's tests are now run against 3.6-dev and all passing. Nice! Getting used to the C Future ;-) On 13 October 2016 at 06:37, INADA Naoki wrote: > Thanks, Ben. > > Both are very helpful information! > > On Thu, Oct 13, 2016 at 2:25 PM, Ben Darnell wrote: > > [+async-sig at python.org, which is the new home for these kinds of > > discussions] > > > > Tornado's tests are now failing on nightly with "TypeError: can't send > > non-None value to a FutureIter": > > https://travis-ci.org/tornadoweb/tornado/jobs/167252979 > > > > -Ben > > > > On Tue, Oct 11, 2016 at 3:55 PM INADA Naoki > wrote: > >> > >> > If you have asyncio based project, and it uses Travis-CI, > >> > >> > please add "nightly" to your .travis.cnf [2]. > >> > >> > > >> > >> > [2] > >> > >> > > >> > https://docs.travis-ci.com/user/languages/python/# > Choosing-Python-versions-to-test-against > >> > >> > >> > >> Travis changed the "nightly" version to 3.7 > >> > >> Now "3.6-dev" is for Python 3.6beta (Still 3.6b1, it may be upgraded > >> soon). > >> > >> > >> > >> -- > >> > >> INADA Naoki > >> > > > > > > -- > INADA Naoki > -- http://lucasbardella.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luca.sbardella at gmail.com Fri Nov 18 17:53:50 2016 From: luca.sbardella at gmail.com (Luca Sbardella) Date: Fri, 18 Nov 2016 22:53:50 +0000 Subject: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future In-Reply-To: References: <408d5393-82e4-4176-8d37-5c3d086c4087@googlegroups.com> Message-ID: But tests taking 1.48 longer to run on average! Anything I should know about 3.6 and performance? On 18 November 2016 at 22:42, Luca Sbardella wrote: > Pulsar's tests are now run against 3.6-dev and all passing. > Nice! > Getting used to the C Future ;-) > > On 13 October 2016 at 06:37, INADA Naoki wrote: > >> Thanks, Ben. >> >> Both are very helpful information! >> >> On Thu, Oct 13, 2016 at 2:25 PM, Ben Darnell wrote: >> > [+async-sig at python.org, which is the new home for these kinds of >> > discussions] >> > >> > Tornado's tests are now failing on nightly with "TypeError: can't send >> > non-None value to a FutureIter": >> > https://travis-ci.org/tornadoweb/tornado/jobs/167252979 >> > >> > -Ben >> > >> > On Tue, Oct 11, 2016 at 3:55 PM INADA Naoki >> wrote: >> >> >> >> > If you have asyncio based project, and it uses Travis-CI, >> >> >> >> > please add "nightly" to your .travis.cnf [2]. >> >> >> >> > >> >> >> >> > [2] >> >> >> >> > >> >> > https://docs.travis-ci.com/user/languages/python/#Choosing- >> Python-versions-to-test-against >> >> >> >> >> >> >> >> Travis changed the "nightly" version to 3.7 >> >> >> >> Now "3.6-dev" is for Python 3.6beta (Still 3.6b1, it may be upgraded >> >> soon). >> >> >> >> >> >> >> >> -- >> >> >> >> INADA Naoki >> >> >> > >> >> >> >> -- >> INADA Naoki >> > > > > -- > http://lucasbardella.com > -- http://lucasbardella.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luca.sbardella at gmail.com Fri Nov 18 18:16:24 2016 From: luca.sbardella at gmail.com (Luca Sbardella) Date: Fri, 18 Nov 2016 23:16:24 +0000 Subject: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future In-Reply-To: References: <408d5393-82e4-4176-8d37-5c3d086c4087@googlegroups.com> Message-ID: > > > > That shouldn?t happen. Are you sure you aren?t running them in debug > mode? Try to comment out imports of ?_asyncio? in futures.py and tasks.py > and run benchmarks in 3.6 to compare Py Futures to C Futures. > > Also, which Python 3.6 version are you using? Please try to build one > from the repo, I?ve fixed a couple of bugs since 3.6b2. > > Running 3.6-dev from travis, currently b4+. No debug mode, that is an order of magnitude slower ;-) Having said that, running another test suite, more asyncio centric, I get the 3.6-dev running at 68% speed v 3.5.2. So it looks very good indeed! Pulsar test suite is large and does not represent asyncio 1:1. Sorry for confusion, I will investigate further -- http://lucasbardella.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From luca.sbardella at gmail.com Sat Nov 19 06:51:20 2016 From: luca.sbardella at gmail.com (Luca Sbardella) Date: Sat, 19 Nov 2016 11:51:20 +0000 Subject: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future In-Reply-To: <04164175-A2AF-437C-A701-735CC49F33EE@gmail.com> References: <408d5393-82e4-4176-8d37-5c3d086c4087@googlegroups.com> <04164175-A2AF-437C-A701-735CC49F33EE@gmail.com> Message-ID: On 18 November 2016 at 23:09, Yury Selivanov wrote: > Also, are you using uvloop or vanilla asyncio? Try to benchmark vanilla > first. And if you have time, please try to test different combinations on > vanilla asyncio: > > Python 3.5 + vanilla asyncio > Python 3.6 + vanilla asyncio > Python 3.6 + Py Future + Py Task > Python 3.6 + Py Future + C Task > Python 3.6 + C Future + C Task > Python 3.6 + Py Future + Py Task > These are results of an asyncio-based benchmark - no sockets involved - highly nested coroutines The --io uv flag indicates a run with uvloop. py35 setup.py bench -a coroutine TestCoroutine.test_coroutine: repeated 10(x1000) times, average 0.50100 secs, stdev 1.13 % py35 setup.py bench -a coroutine --io uv TestCoroutine.test_coroutine: repeated 10(x1000) times, average 0.08246 secs, stdev 3.87 % py36 setup.py bench -a coroutine TestCoroutine.test_coroutine: repeated 10(x1000) times, average 0.26563 secs, stdev 2.35 % py36 setup.py bench -a coroutine --io uv TestCoroutine.test_coroutine: repeated 10(x1000) times, average 0.06762 secs, stdev 5.43 % I'll benchmark with sockets next -------------- next part -------------- An HTML attachment was scrubbed... URL: From vxgmichel at gmail.com Mon Nov 21 12:31:40 2016 From: vxgmichel at gmail.com (Vincent Michel) Date: Mon, 21 Nov 2016 18:31:40 +0100 Subject: [Async-sig] aiostream release Message-ID: <19e2204c-9168-d55c-2c44-ab0464856978@gmail.com> Hi all, I just released aiostream, a collection of generator-based operators for asynchronous iteration. - GitHub: https://github.com/vxgmichel/aiostream - PyPI: https://pypi.org/project/aiostream - Docs: http://pythonhosted.org/aiostream It can be seen as an asynchronous version of itertools, although some aspects are slightly different. The stream operators rely heavily on asynchronous generators (PEP 525), so python 3.6 is required. The project is fully tested, and the documentation provides a couple of examples: http://pythonhosted.org/aiostream/examples.html Cheers, /Vincent From alex.gronholm at nextday.fi Tue Nov 22 09:22:37 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Tue, 22 Nov 2016 16:22:37 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: Message-ID: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> Does this library offer any advantages over the same functionality in asyncio_extras? 15.11.2016, 08:52, Nathaniel Smith kirjoitti: > Hi all, > > I just released v1.2 of my async_generator package: > > https://pypi.org/pypi/async_generator/ > > This package makes it easy to write PEP 525-style "async generators", > even on Python 3.5. It looks like: > > from async_generator import async_generator, yield_, yield_from_ > > @async_generator > async def parse_json_separated_newlines_from_network(streamreader): > async for line in streamreader: > await yield_(json.loads(line)) > > Starting in v1.1 (released last week but not announced widely), the > generators are now highly compatible with 3.6's native async > generators, including full support for athrow / asend / aclose. They > also fully support "yield from", which is otherwise unavailable even > on Python 3.6. > > v1.2 fixed some edge cases in "yield from" -- but the headline feature > is that we're now *so* compatible with 3.6's native async generators > that you can now use our yield from support directly inside a native > generator: > > from async_generator import yield_from_ > > async def f(): > yield 2 > yield 3 > > # yields [1, 2, 3, 4] > async def g(): > yield 1 > await yield_from_(f()) > yield 4 > > Have fun, > -n > From njs at pobox.com Tue Nov 22 15:32:45 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Nov 2016 12:32:45 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> Message-ID: On Nov 22, 2016 6:22 AM, "Alex Gr?nholm" wrote: > > Does this library offer any advantages over the same functionality in asyncio_extras? You would probably know better than me, but last time I looked, the generator code in asyncio_extras didn't support asend/athrow/aclose, didn't support yield from, and took shortcuts that assume that you're using asyncio as your coroutine runner. So I believe async_generator is more complete and more correct (in the sense that it sticks strictly to implementing the language level semantics for async generators without assuming any particular runner). Of course asyncio_extras has lots of other handy things in it too, while async_generator is very strictly focused on just this one thing. (Well, I threw in an implementation of aclosing too, since you can't really use async generators without it, but that's pretty trivial.) It might make sense for asyncio_extras to out-source generator handling to async_generator? Or not. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Tue Nov 22 17:22:52 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Wed, 23 Nov 2016 00:22:52 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> Message-ID: <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> I'm not sure where asyncio_extras's async generator implementation assumes you're using asyncio. Could you elaborate on that? Native async generators don't support "yield from" either, so I didn't try to add that until there's a new PEP that defines its exact semantics. As for aclose/asend/athrow support, I will have to look into that. Could we synchronize the implementations so that they're compatible with each other? I would've preferred there to be only a single implementation (asyncio_extras was published first, but admittedly I didn't advertise it that well) but now it'd be best if we work together, wouldn't you agree? 22.11.2016, 22:32, Nathaniel Smith kirjoitti: > > On Nov 22, 2016 6:22 AM, "Alex Gr?nholm" > wrote: > > > > Does this library offer any advantages over the same functionality > in asyncio_extras? > > You would probably know better than me, but last time I looked, the > generator code in asyncio_extras didn't support asend/athrow/aclose, > didn't support yield from, and took shortcuts that assume that you're > using asyncio as your coroutine runner. So I believe async_generator > is more complete and more correct (in the sense that it sticks > strictly to implementing the language level semantics for async > generators without assuming any particular runner). > > Of course asyncio_extras has lots of other handy things in it too, > while async_generator is very strictly focused on just this one thing. > (Well, I threw in an implementation of aclosing too, since you can't > really use async generators without it, but that's pretty trivial.) It > might make sense for asyncio_extras to out-source generator handling > to async_generator? Or not. > > -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Nov 22 18:34:43 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Nov 2016 15:34:43 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> Message-ID: On Tue, Nov 22, 2016 at 2:22 PM, Alex Gr?nholm wrote: > I'm not sure where asyncio_extras's async generator implementation assumes > you're using asyncio. Could you elaborate on that? If I'm reading it right, it assumes that the only two things that might be yielded to the coroutine runner are either (a) the special yield wrapper, or (b) an awaitable object like an asyncio.Future. This works on asyncio, because that's all the asyncio runner supports, but it doesn't work with (for example) curio. async_generator (like native async generators) allows arbitrary objects to be yielded to the coroutine runner. > Native async generators don't support "yield from" either, so I didn't try > to add that until there's a new PEP that defines its exact semantics. Yeah, one of the motivations for async_generator has been to figure out what the semantics for native generators should be :-). The native async generators in 3.6 use the same implementation trick as async_generator (not entirely sure if Yury got it from me, or if it's an independent invention), and the yield from implementation is partly motivated as a proof of concept / testing ground for hopefully getting it into 3.7. The yield from semantics are pretty straightforward though, I think. > As for > aclose/asend/athrow support, I will have to look into that. > > Could we synchronize the implementations so that they're compatible with > each other? I would've preferred there to be only a single implementation > (asyncio_extras was published first, but admittedly I didn't advertise it > that well) but now it'd be best if we work together, wouldn't you agree? Makes sense to me :-). I think the obvious approach would be for async_extras to just depend on async_generator, since AFAICT async_generator is pretty much complete, and like you say, there's not much point in carrying around two copies of the same thing. But if you have another suggestion I'd be interested to hear it... -n > > 22.11.2016, 22:32, Nathaniel Smith kirjoitti: > > On Nov 22, 2016 6:22 AM, "Alex Gr?nholm" wrote: >> >> Does this library offer any advantages over the same functionality in >> asyncio_extras? > > You would probably know better than me, but last time I looked, the > generator code in asyncio_extras didn't support asend/athrow/aclose, didn't > support yield from, and took shortcuts that assume that you're using asyncio > as your coroutine runner. So I believe async_generator is more complete and > more correct (in the sense that it sticks strictly to implementing the > language level semantics for async generators without assuming any > particular runner). > > Of course asyncio_extras has lots of other handy things in it too, while > async_generator is very strictly focused on just this one thing. (Well, I > threw in an implementation of aclosing too, since you can't really use async > generators without it, but that's pretty trivial.) It might make sense for > asyncio_extras to out-source generator handling to async_generator? Or not. > > -n > > -- Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From dahalpi at gmail.com Wed Nov 23 09:53:53 2016 From: dahalpi at gmail.com (David Pineda) Date: Wed, 23 Nov 2016 11:53:53 -0300 Subject: [Async-sig] A tasks list and run every element again when finish Message-ID: Hello. I'm using the asyncio module. I have a list with different task, every tasks has different time to be done. In a figure: fn 1 -> -> -> -> fn 2 --> --> --> --> --> . . . fn N -----> -----> -----> -----> So i need to repeat every task when its done. I tried to use 'while' but while have to complete all list first to repeat the list, this is not useful for me. Y ask you, fellows, about if there are a way to do that. In particular these tasks are with socket and communication methods. I ve tried with multiprocess but the broke because are different loop events. Yes, i tryyo mix asyncio with multiprocess, but in this case didn't work I will be very thankfull of you. Best Regards! -- David A. Pineda Osorio F:+56 9 82142267 Ingeniero Civil Electricista Universidad de Chile -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Thu Nov 24 02:29:51 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Thu, 24 Nov 2016 09:29:51 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> Message-ID: <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> 23.11.2016, 01:34, Nathaniel Smith kirjoitti: > > On Tue, Nov 22, 2016 at 2:22 PM, Alex Gr?nholm > > wrote: > > I'm not sure where asyncio_extras's async generator implementation > assumes > > you're using asyncio. Could you elaborate on that? > > If I'm reading it right, it assumes that the only two things that > might be yielded to the coroutine runner are either (a) the special > yield wrapper, or (b) an awaitable object like an asyncio.Future. This > works on asyncio, because that's all the asyncio runner supports, but > it doesn't work with (for example) curio. async_generator (like native > async generators) allows arbitrary objects to be yielded to the > coroutine runner. > You are misreading the code. It is in no way limited to what asyncio accepts. It doesn't even import asyncio in the asyncyield or generator modules. The only parts of the library that depend on PEP 3156 event loops are the ones that involve executors and threads. > > > Native async generators don't support "yield from" either, so I > didn't try > > to add that until there's a new PEP that defines its exact semantics. > > Yeah, one of the motivations for async_generator has been to figure > out what the semantics for native generators should be :-). The native > async generators in 3.6 use the same implementation trick as > async_generator (not entirely sure if Yury got it from me, or if it's > an independent invention), and the yield from implementation is partly > motivated as a proof of concept / testing ground for hopefully getting > it into 3.7. The yield from semantics are pretty straightforward > though, I think. > > > As for > > aclose/asend/athrow support, I will have to look into that. > > > > Could we synchronize the implementations so that they're compatible with > > each other? I would've preferred there to be only a single > implementation > > (asyncio_extras was published first, but admittedly I didn't > advertise it > > that well) but now it'd be best if we work together, wouldn't you agree? > > Makes sense to me :-). I think the obvious approach would be for > async_extras to just depend on async_generator, since AFAICT > async_generator is pretty much complete, and like you say, there's not > much point in carrying around two copies of the same thing. But if you > have another suggestion I'd be interested to hear it... > I may consider that at some point, but I'm not yet comfortable with those extended capabilities. Meanwhile I will add any missing functionality (asend, athrow etc.) to mine. One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching. > > -n > > > > > 22.11.2016, 22:32, Nathaniel Smith kirjoitti: > > > > On Nov 22, 2016 6:22 AM, "Alex Gr?nholm" > wrote: > >> > >> Does this library offer any advantages over the same functionality in > >> asyncio_extras? > > > > You would probably know better than me, but last time I looked, the > > generator code in asyncio_extras didn't support asend/athrow/aclose, > didn't > > support yield from, and took shortcuts that assume that you're using > asyncio > > as your coroutine runner. So I believe async_generator is more > complete and > > more correct (in the sense that it sticks strictly to implementing the > > language level semantics for async generators without assuming any > > particular runner). > > > > Of course asyncio_extras has lots of other handy things in it too, while > > async_generator is very strictly focused on just this one thing. > (Well, I > > threw in an implementation of aclosing too, since you can't really > use async > > generators without it, but that's pretty trivial.) It might make > sense for > > asyncio_extras to out-source generator handling to async_generator? > Or not. > > > > -n > > > > > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Nov 24 16:23:06 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Nov 2016 13:23:06 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: On Nov 23, 2016 11:29 PM, "Alex Gr?nholm" wrote: > > 23.11.2016, 01:34, Nathaniel Smith kirjoitti: >> >> On Tue, Nov 22, 2016 at 2:22 PM, Alex Gr?nholm wrote: >> > I'm not sure where asyncio_extras's async generator implementation assumes >> > you're using asyncio. Could you elaborate on that? >> >> If I'm reading it right, it assumes that the only two things that might be yielded to the coroutine runner are either (a) the special yield wrapper, or (b) an awaitable object like an asyncio.Future. This works on asyncio, because that's all the asyncio runner supports, but it doesn't work with (for example) curio. async_generator (like native async generators) allows arbitrary objects to be yielded to the coroutine runner. > > You are misreading the code. It is in no way limited to what asyncio accepts. It doesn't even import asyncio in the asyncyield or generator modules. The only parts of the library that depend on PEP 3156 event loops are the ones that involve executors and threads. I didn't say that it imported asyncio. I said that it assumes the only things that will be yielded are the things that asyncio yields. This is the line that I'm worried about: https://github.com/agronholm/asyncio_extras/blob/aec412e1b7034ca3cad386c381e655ce3547fee3/asyncio_extras/asyncyield.py#L40 The code awaits the value yielded by the coroutine, but there's no guarantee that this value is awaitable. It's an arbitrary Python object representing a message sent to the coroutine runner. It turns out that asyncio only uses awaitable objects for its messages, so this code can get away with this on asyncio, but if you try using this code with curio then I'm pretty sure you're going to end up doing something like "await (3,)" and then blowing up. >> > Native async generators don't support "yield from" either, so I didn't try >> > to add that until there's a new PEP that defines its exact semantics. >> >> Yeah, one of the motivations for async_generator has been to figure out what the semantics for native generators should be :-). The native async generators in 3.6 use the same implementation trick as async_generator (not entirely sure if Yury got it from me, or if it's an independent invention), and the yield from implementation is partly motivated as a proof of concept / testing ground for hopefully getting it into 3.7. The yield from semantics are pretty straightforward though, I think. >> >> > As for >> > aclose/asend/athrow support, I will have to look into that. >> > >> > Could we synchronize the implementations so that they're compatible with >> > each other? I would've preferred there to be only a single implementation >> > (asyncio_extras was published first, but admittedly I didn't advertise it >> > that well) but now it'd be best if we work together, wouldn't you agree? >> >> Makes sense to me :-). I think the obvious approach would be for async_extras to just depend on async_generator, since AFAICT async_generator is pretty much complete, and like you say, there's not much point in carrying around two copies of the same thing. But if you have another suggestion I'd be interested to hear it... > > I may consider that at some point, but I'm not yet comfortable with those extended capabilities. Meanwhile I will add any missing functionality (asend, athrow etc.) to mine. You could just not import yield_from_ and then you don't have any extended capabilities... but up to you, obviously. > One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching. Good point. That should be pretty trivial to add. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Nov 25 02:25:32 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 24 Nov 2016 23:25:32 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith wrote: [...] >> One thing I noticed is that there seems to be no way to detect async >> generator functions in your implementation. That is something I would want >> to have before switching. > > Good point. That should be pretty trivial to add. Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with collections.abc.AsyncGenerator. -n -- Nathaniel J. Smith -- https://vorpus.org From alex.gronholm at nextday.fi Fri Nov 25 03:03:49 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Fri, 25 Nov 2016 10:03:49 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: <68c1abe5-7df7-6cba-1f56-ed697f1c721b@nextday.fi> 24.11.2016, 23:23, Nathaniel Smith kirjoitti: > > On Nov 23, 2016 11:29 PM, "Alex Gr?nholm" > wrote: > > > > 23.11.2016, 01:34, Nathaniel Smith kirjoitti: > >> > >> On Tue, Nov 22, 2016 at 2:22 PM, Alex Gr?nholm > > wrote: > >> > I'm not sure where asyncio_extras's async generator > implementation assumes > >> > you're using asyncio. Could you elaborate on that? > >> > >> If I'm reading it right, it assumes that the only two things that > might be yielded to the coroutine runner are either (a) the special > yield wrapper, or (b) an awaitable object like an asyncio.Future. This > works on asyncio, because that's all the asyncio runner supports, but > it doesn't work with (for example) curio. async_generator (like native > async generators) allows arbitrary objects to be yielded to the > coroutine runner. > > > > You are misreading the code. It is in no way limited to what asyncio > accepts. It doesn't even import asyncio in the asyncyield or generator > modules. The only parts of the library that depend on PEP 3156 event > loops are the ones that involve executors and threads. > > I didn't say that it imported asyncio. I said that it assumes the only > things that will be yielded are the things that asyncio yields. This > is the line that I'm worried about: > > https://github.com/agronholm/asyncio_extras/blob/aec412e1b7034ca3cad386c381e655ce3547fee3/asyncio_extras/asyncyield.py#L40 > > The code awaits the value yielded by the coroutine, but there's no > guarantee that this value is awaitable. It's an arbitrary Python > object representing a message sent to the coroutine runner. It turns > out that asyncio only uses awaitable objects for its messages, so this > code can get away with this on asyncio, but if you try using this code > with curio then I'm pretty sure you're going to end up doing something > like "await (3,)" and then blowing up. > PEP 492 clearly states the following: It is aTypeErrorto pass anything other than an/awaitable/object to anawaitexpression. That (3,) is not an awaitable, so the example is invalid. That said, I will re-examine this part of the implementation and correct it if necessary. So far I just haven't encountered anything that would produce an error. > >> > Native async generators don't support "yield from" either, so I > didn't try > >> > to add that until there's a new PEP that defines its exact semantics. > >> > >> Yeah, one of the motivations for async_generator has been to figure > out what the semantics for native generators should be :-). The native > async generators in 3.6 use the same implementation trick as > async_generator (not entirely sure if Yury got it from me, or if it's > an independent invention), and the yield from implementation is partly > motivated as a proof of concept / testing ground for hopefully getting > it into 3.7. The yield from semantics are pretty straightforward > though, I think. > >> > >> > As for > >> > aclose/asend/athrow support, I will have to look into that. > >> > > >> > Could we synchronize the implementations so that they're > compatible with > >> > each other? I would've preferred there to be only a single > implementation > >> > (asyncio_extras was published first, but admittedly I didn't > advertise it > >> > that well) but now it'd be best if we work together, wouldn't you > agree? > >> > >> Makes sense to me :-). I think the obvious approach would be for > async_extras to just depend on async_generator, since AFAICT > async_generator is pretty much complete, and like you say, there's not > much point in carrying around two copies of the same thing. But if you > have another suggestion I'd be interested to hear it... > > > > I may consider that at some point, but I'm not yet comfortable with > those extended capabilities. Meanwhile I will add any missing > functionality (asend, athrow etc.) to mine. > > You could just not import yield_from_ and then you don't have any > extended capabilities... but up to you, obviously. > > > One thing I noticed is that there seems to be no way to detect async > generator functions in your implementation. That is something I would > want to have before switching. > > Good point. That should be pretty trivial to add. > > -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Fri Nov 25 02:59:13 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Fri, 25 Nov 2016 09:59:13 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: 25.11.2016, 09:25, Nathaniel Smith kirjoitti: > On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith wrote: > [...] >>> One thing I noticed is that there seems to be no way to detect async >>> generator functions in your implementation. That is something I would want >>> to have before switching. >> Good point. That should be pretty trivial to add. > Just pushed async_generator v1.3 to PyPI, with new isasyncgen, > isasyncgenfunction, and (on 3.6) registration with > collections.abc.AsyncGenerator. And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating? > > -n > From njs at pobox.com Fri Nov 25 05:09:59 2016 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Nov 2016 02:09:59 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: On Thu, Nov 24, 2016 at 11:59 PM, Alex Gr?nholm wrote: > 25.11.2016, 09:25, Nathaniel Smith kirjoitti: >> >> On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith wrote: >> [...] >>>> >>>> One thing I noticed is that there seems to be no way to detect async >>>> generator functions in your implementation. That is something I would >>>> want >>>> to have before switching. >>> >>> Good point. That should be pretty trivial to add. >> >> Just pushed async_generator v1.3 to PyPI, with new isasyncgen, >> isasyncgenfunction, and (on 3.6) registration with >> collections.abc.AsyncGenerator. > > And you just *had* to make it incompatible with the async generators from > asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? > I thought we agreed on cooperating? I started doing that, but... it's not an async generator, isasyncgen is a different inspect function... -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Fri Nov 25 05:14:34 2016 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Nov 2016 02:14:34 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: <68c1abe5-7df7-6cba-1f56-ed697f1c721b@nextday.fi> References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> <68c1abe5-7df7-6cba-1f56-ed697f1c721b@nextday.fi> Message-ID: On Fri, Nov 25, 2016 at 12:03 AM, Alex Gr?nholm wrote: > 24.11.2016, 23:23, Nathaniel Smith kirjoitti: > > On Nov 23, 2016 11:29 PM, "Alex Gr?nholm" wrote: >> >> 23.11.2016, 01:34, Nathaniel Smith kirjoitti: >>> >>> On Tue, Nov 22, 2016 at 2:22 PM, Alex Gr?nholm >>> wrote: >>> > I'm not sure where asyncio_extras's async generator implementation >>> > assumes >>> > you're using asyncio. Could you elaborate on that? >>> >>> If I'm reading it right, it assumes that the only two things that might >>> be yielded to the coroutine runner are either (a) the special yield wrapper, >>> or (b) an awaitable object like an asyncio.Future. This works on asyncio, >>> because that's all the asyncio runner supports, but it doesn't work with >>> (for example) curio. async_generator (like native async generators) allows >>> arbitrary objects to be yielded to the coroutine runner. >> >> You are misreading the code. It is in no way limited to what asyncio >> accepts. It doesn't even import asyncio in the asyncyield or generator >> modules. The only parts of the library that depend on PEP 3156 event loops >> are the ones that involve executors and threads. > > I didn't say that it imported asyncio. I said that it assumes the only > things that will be yielded are the things that asyncio yields. This is the > line that I'm worried about: > > > https://github.com/agronholm/asyncio_extras/blob/aec412e1b7034ca3cad386c381e655ce3547fee3/asyncio_extras/asyncyield.py#L40 > > The code awaits the value yielded by the coroutine, but there's no guarantee > that this value is awaitable. It's an arbitrary Python object representing a > message sent to the coroutine runner. It turns out that asyncio only uses > awaitable objects for its messages, so this code can get away with this on > asyncio, but if you try using this code with curio then I'm pretty sure > you're going to end up doing something like "await (3,)" and then blowing > up. > > PEP 492 clearly states the following: > > It is a TypeError to pass anything other than an awaitable object to an > await expression. > > That (3,) is not an awaitable, so the example is invalid. That said, I will > re-examine this part of the implementation and correct it if necessary. I feel like I'm running out of ideas for how to explain this, and starting to just repeat myself :-(. There is nothing that says the return value from coro.__next__() must be awaitable. Coroutines can yield arbitrary objects. (3,) is not awaitable, but it's a perfectly valid thing to be yielded from a coroutine. > So far I just haven't encountered anything that would produce an error. That's because you've only tested on asyncio, not curio. I promise! -n -- Nathaniel J. Smith -- https://vorpus.org From dahalpi at gmail.com Fri Nov 25 12:49:07 2016 From: dahalpi at gmail.com (David Pineda) Date: Fri, 25 Nov 2016 14:49:07 -0300 Subject: [Async-sig] A tasks list and run every element again when finish In-Reply-To: References: Message-ID: I answer myself. Mabe it's useful for the DOCUMENTATION: SIMPLE SOL: import asyncio async def holacoro(): print("Hola %d" % 1) await asyncio.sleep(1) def renew(*args): task=loop.create_task(holacoro()) task.add_done_callback(renew) task=loop.create_task(holacoro()) task.add_done_callback(renew) loop=asyncio.get_event_loop() try: loop.run_forever() except KeyboardInterrupt: print('Loop stopped') BETTER SOL: import asyncio import functools async def holacoro(v): print("Hola %d" % v) await asyncio.sleep(1) return v+1 #special for every coroutine async def coromask(coro, args): result=await coro(*args) return [result] def renew(task, coromask, coro, *args): print(task.result()) task=loop.create_task(coromask(coro,task.result())) task.add_done_callback(functools.partial(renew, task, coromask, coro)) args=[1] task=loop.create_task(coromask(holacoro,args)) task.add_done_callback(functools.partial(renew, task, coromask, holacoro)) loop=asyncio.get_event_loop() try: loop.run_forever() except KeyboardInterrupt: print('Loop stopped') BETTER WITH 2 CORO: import asyncio import functools async def holacoro(v): print("Hola %d" % v) await asyncio.sleep(1) return v+1 async def sumacoro(*args): c=sum(args) print("La suma es %d" %c) await asyncio.sleep(3) return c #special for every coroutine async def coromask(coro, args): result=await coro(*args) return [result] #special for every coroutine async def coromask_suma(coro, args): _in=[args[-1]] result=await coro(*args) _in.append(result) return _in def renew(task, mask, coro, *args): result=task.result() task=loop.create_task(mask(coro,result)) task.add_done_callback(functools.partial(renew, task, mask, coro)) loop=asyncio.get_event_loop() args=[1] task1=loop.create_task(coromask(holacoro,args)) task1.add_done_callback(functools.partial(renew, task1, coromask, holacoro)) args2=[1,2] task2=loop.create_task(coromask_suma(sumacoro,args2)) task2.add_done_callback(functools.partial(renew, task2, coromask_suma, sumacoro)) try: loop.run_forever() except KeyboardInterrupt: print('Loop stopped') 2016-11-23 11:53 GMT-03:00 David Pineda : > Hello. > > I'm using the asyncio module. I have a list with different task, every > tasks has different time to be done. > > In a figure: > > fn 1 -> -> -> -> > fn 2 --> --> --> --> --> > . > . > . > fn N -----> -----> -----> -----> > > So i need to repeat every task when its done. > > I tried to use 'while' but while have to complete all list first to repeat > the list, this is not useful for me. > > Y ask you, fellows, about if there are a way to do that. In particular > these tasks are with socket and communication methods. I ve tried > with multiprocess but the broke because are different loop events. Yes, i > tryyo mix asyncio with multiprocess, but in this case didn't work > > I will be very thankfull of you. > > Best Regards! > > > -- > David A. Pineda Osorio > F:+56 9 82142267 > Ingeniero Civil Electricista > Universidad de Chile > -- David A. Pineda Osorio F:+56 9 82142267 Ingeniero Civil Electricista Universidad de Chile -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Fri Nov 25 13:46:12 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Fri, 25 Nov 2016 20:46:12 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: 25.11.2016, 12:09, Nathaniel Smith kirjoitti: > On Thu, Nov 24, 2016 at 11:59 PM, Alex Gr?nholm > wrote: >> 25.11.2016, 09:25, Nathaniel Smith kirjoitti: >>> On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith wrote: >>> [...] >>>>> One thing I noticed is that there seems to be no way to detect async >>>>> generator functions in your implementation. That is something I would >>>>> want >>>>> to have before switching. >>>> Good point. That should be pretty trivial to add. >>> Just pushed async_generator v1.3 to PyPI, with new isasyncgen, >>> isasyncgenfunction, and (on 3.6) registration with >>> collections.abc.AsyncGenerator. >> And you just *had* to make it incompatible with the async generators from >> asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? >> I thought we agreed on cooperating? > I started doing that, but... it's not an async generator, isasyncgen > is a different inspect function... > > -n > It's an arbitrary string that will likely never be seen by anyone except for people working on the libraries. But it makes a world of difference in compatibility. I named it this way to be consistent with asyncio which marks yield-based coroutines with "_is_coroutine". So please reconsider. From njs at pobox.com Sat Nov 26 02:47:22 2016 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 25 Nov 2016 23:47:22 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: On Fri, Nov 25, 2016 at 10:46 AM, Alex Gr?nholm wrote: > 25.11.2016, 12:09, Nathaniel Smith kirjoitti: >> >> On Thu, Nov 24, 2016 at 11:59 PM, Alex Gr?nholm >> wrote: >>> >>> 25.11.2016, 09:25, Nathaniel Smith kirjoitti: >>>> >>>> On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith wrote: >>>> [...] >>>>>> >>>>>> One thing I noticed is that there seems to be no way to detect async >>>>>> generator functions in your implementation. That is something I would >>>>>> want >>>>>> to have before switching. >>>>> >>>>> Good point. That should be pretty trivial to add. >>>> >>>> Just pushed async_generator v1.3 to PyPI, with new isasyncgen, >>>> isasyncgenfunction, and (on 3.6) registration with >>>> collections.abc.AsyncGenerator. >>> >>> And you just *had* to make it incompatible with the async generators from >>> asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? >>> I thought we agreed on cooperating? >> >> I started doing that, but... it's not an async generator, isasyncgen >> is a different inspect function... >> >> -n >> > It's an arbitrary string that will likely never be seen by anyone except for > people working on the libraries. But it makes a world of difference in > compatibility. I named it this way to be consistent with asyncio which marks > yield-based coroutines with "_is_coroutine". So please reconsider. Sure, I guess it doesn't matter too much either way. Can we pause a moment to ask whether we really want the two async generator types to return true for each other's introspection function, though? Right now they aren't really interchangeable, and the only user for this kind of introspection I've seen is your contextmanager.py. It seems to use the inspect versions of isasyncgen and isasyncgenfunction, because it wants to know whether asend/athrow/aclose are supported. Right now, if it switched to using async_generator's isasyncgen/isasyncgenfunction, it would continue to work; but if I then switched async_generator's isasyncgen/isasyncgenfunction to detect asyncio_extras's generators, it would stop working again. Speaking of which, what do you want to do about isasyncgen? -n -- Nathaniel J. Smith -- https://vorpus.org From alex.gronholm at nextday.fi Sat Nov 26 05:07:15 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Sat, 26 Nov 2016 12:07:15 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> Message-ID: <5c9f1648-4f47-bf7d-77d9-340d7d523234@nextday.fi> 26.11.2016, 09:47, Nathaniel Smith kirjoitti: > On Fri, Nov 25, 2016 at 10:46 AM, Alex Gr?nholm > wrote: >> 25.11.2016, 12:09, Nathaniel Smith kirjoitti: >>> On Thu, Nov 24, 2016 at 11:59 PM, Alex Gr?nholm >>> wrote: >>>> 25.11.2016, 09:25, Nathaniel Smith kirjoitti: >>>>> On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith wrote: >>>>> [...] >>>>>>> One thing I noticed is that there seems to be no way to detect async >>>>>>> generator functions in your implementation. That is something I would >>>>>>> want >>>>>>> to have before switching. >>>>>> Good point. That should be pretty trivial to add. >>>>> Just pushed async_generator v1.3 to PyPI, with new isasyncgen, >>>>> isasyncgenfunction, and (on 3.6) registration with >>>>> collections.abc.AsyncGenerator. >>>> And you just *had* to make it incompatible with the async generators from >>>> asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? >>>> I thought we agreed on cooperating? >>> I started doing that, but... it's not an async generator, isasyncgen >>> is a different inspect function... >>> >>> -n >>> >> It's an arbitrary string that will likely never be seen by anyone except for >> people working on the libraries. But it makes a world of difference in >> compatibility. I named it this way to be consistent with asyncio which marks >> yield-based coroutines with "_is_coroutine". So please reconsider. > Sure, I guess it doesn't matter too much either way. > > Can we pause a moment to ask whether we really want the two async > generator types to return true for each other's introspection > function, though? Right now they aren't really interchangeable, and > the only user for this kind of introspection I've seen is your > contextmanager.py. It seems to use the inspect versions of isasyncgen > and isasyncgenfunction, because it wants to know whether > asend/athrow/aclose are supported. Right now, if it switched to using > async_generator's isasyncgen/isasyncgenfunction, it would continue to > work; but if I then switched async_generator's > isasyncgen/isasyncgenfunction to detect asyncio_extras's generators, > it would stop working again. > > Speaking of which, what do you want to do about isasyncgen? > > -n > I've created a new branch in asyncio_extras which delegates the implementation of async generators to async_generator. All tests pass with 100% combined coverage. Before I merge this into master, I would like to implement some changes in your library: * Set the default value for yield_() argument to None, as in my yield_async() * Add type hints to methods and functions * Look into implementing missing attributes which neither lib had before (if technically possible): ag_frame, ag_running, ag_code, __name__, __qualname__ I hope these changes aren't too invasive. They should be fully backwards compatible. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Nov 26 10:42:59 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 26 Nov 2016 07:42:59 -0800 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: <5c9f1648-4f47-bf7d-77d9-340d7d523234@nextday.fi> References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> <5c9f1648-4f47-bf7d-77d9-340d7d523234@nextday.fi> Message-ID: On Nov 26, 2016 2:07 AM, "Alex Gr?nholm" wrote: > > 26.11.2016, 09:47, Nathaniel Smith kirjoitti: >> >> On Fri, Nov 25, 2016 at 10:46 AM, Alex Gr?nholm wrote: >>> >>> 25.11.2016, 12:09, Nathaniel Smith kirjoitti: >>>> >>>> On Thu, Nov 24, 2016 at 11:59 PM, Alex Gr?nholm < alex.gronholm at nextday.fi> wrote: >>>>> >>>>> 25.11.2016, 09:25, Nathaniel Smith kirjoitti: >>>>>> >>>>>> On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith wrote: [...] >>>>>>>> >>>>>>>> One thing I noticed is that there seems to be no way to detect async generator functions in your implementation. That is something I would want to have before switching. >>>>>>> >>>>>>> Good point. That should be pretty trivial to add. >>>>>> >>>>>> Just pushed async_generator v1.3 to PyPI, with new isasyncgen, isasyncgenfunction, and (on 3.6) registration with collections.abc.AsyncGenerator. >>>>> >>>>> And you just *had* to make it incompatible with the async generators from asyncio_extras? "_is_async_gen_function" vs "_is_async_generator"? I thought we agreed on cooperating? >>>> >>>> I started doing that, but... it's not an async generator, isasyncgen is a different inspect function... -n >>> >>> It's an arbitrary string that will likely never be seen by anyone except for people working on the libraries. But it makes a world of difference in compatibility. I named it this way to be consistent with asyncio which marks yield-based coroutines with "_is_coroutine". So please reconsider. >> >> Sure, I guess it doesn't matter too much either way. Can we pause a moment to ask whether we really want the two async generator types to return true for each other's introspection function, though? Right now they aren't really interchangeable, and the only user for this kind of introspection I've seen is your contextmanager.py. It seems to use the inspect versions of isasyncgen and isasyncgenfunction, because it wants to know whether asend/athrow/aclose are supported. Right now, if it switched to using async_generator's isasyncgen/isasyncgenfunction, it would continue to work; but if I then switched async_generator's isasyncgen/isasyncgenfunction to detect asyncio_extras's generators, it would stop working again. Speaking of which, what do you want to do about isasyncgen? -n > > I've created a new branch in asyncio_extras which delegates the implementation of async generators to async_generator. All tests pass with 100% combined coverage. Cool! > Before I merge this into master, I would like to implement some changes in your library: > Set the default value for yield_() argument to None, as in my yield_async() Makes sense. > Add type hints to methods and functions I haven't had a chance to play with these much, but no objections. > Look into implementing missing attributes which neither lib had before (if technically possible): ag_frame, ag_running, ag_code, __name__, __qualname__ __name__ and __qualname__ are obviously a good idea. The others I'm not so sure about -- I'd worry that anyone who's reaching that deep into the internals is not going to be satisfied by any half-way attempts at compatibility, and it seems unlikely we can achieve detailed compatibility at that level. But maybe it's not that bad; I haven't looked at them in detail. Do you have any particular use cases in mind for them? -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Sat Nov 26 11:04:58 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Sat, 26 Nov 2016 18:04:58 +0200 Subject: [Async-sig] [ANN] async_generator v1.2 released In-Reply-To: References: <7486e1a1-6817-b692-0b7e-d62f58314c76@nextday.fi> <5188f86b-6d99-f2e9-88d4-8f2100e9d848@nextday.fi> <6697e0cc-8967-04ba-5650-51ca03cba53d@nextday.fi> <5c9f1648-4f47-bf7d-77d9-340d7d523234@nextday.fi> Message-ID: <3cfd52a0-ed4d-9815-cf4e-14ca81fb5e7e@nextday.fi> 26.11.2016, 17:42, Nathaniel Smith kirjoitti: > > On Nov 26, 2016 2:07 AM, "Alex Gr?nholm" > wrote: > > > > 26.11.2016, 09:47, Nathaniel Smith kirjoitti: > >> > >> On Fri, Nov 25, 2016 at 10:46 AM, Alex Gr?nholm > > wrote: > >>> > >>> 25.11.2016, 12:09, Nathaniel Smith kirjoitti: > >>>> > >>>> On Thu, Nov 24, 2016 at 11:59 PM, Alex Gr?nholm > > wrote: > >>>>> > >>>>> 25.11.2016, 09:25, Nathaniel Smith kirjoitti: > >>>>>> > >>>>>> On Thu, Nov 24, 2016 at 1:23 PM, Nathaniel Smith > > wrote: [...] > >>>>>>>> > >>>>>>>> One thing I noticed is that there seems to be no way to > detect async generator functions in your implementation. That is > something I would want to have before switching. > >>>>>>> > >>>>>>> Good point. That should be pretty trivial to add. > >>>>>> > >>>>>> Just pushed async_generator v1.3 to PyPI, with new isasyncgen, > isasyncgenfunction, and (on 3.6) registration with > collections.abc.AsyncGenerator. > >>>>> > >>>>> And you just *had* to make it incompatible with the async > generators from asyncio_extras? "_is_async_gen_function" vs > "_is_async_generator"? I thought we agreed on cooperating? > >>>> > >>>> I started doing that, but... it's not an async generator, > isasyncgen is a different inspect function... -n > >>> > >>> It's an arbitrary string that will likely never be seen by anyone > except for people working on the libraries. But it makes a world of > difference in compatibility. I named it this way to be consistent with > asyncio which marks yield-based coroutines with "_is_coroutine". So > please reconsider. > >> > >> Sure, I guess it doesn't matter too much either way. Can we pause > a moment to ask whether we really want the two async generator types > to return true for each other's introspection function, though? Right > now they aren't really interchangeable, and the only user for this > kind of introspection I've seen is your contextmanager.py. It seems to > use the inspect versions of isasyncgen and isasyncgenfunction, because > it wants to know whether asend/athrow/aclose are supported. Right now, > if it switched to using async_generator's > isasyncgen/isasyncgenfunction, it would continue to work; but if I > then switched async_generator's isasyncgen/isasyncgenfunction to > detect asyncio_extras's generators, it would stop working again. > Speaking of which, what do you want to do about isasyncgen? -n > > > > I've created a new branch in asyncio_extras which delegates the > implementation of async generators to async_generator. All tests pass > with 100% combined coverage. > > Cool! > > > Before I merge this into master, I would like to implement some > changes in your library: > > Set the default value for yield_() argument to None, as in my > yield_async() > > Makes sense. > > > Add type hints to methods and functions > > I haven't had a chance to play with these much, but no objections. > > > Look into implementing missing attributes which neither lib had > before (if technically possible): ag_frame, ag_running, ag_code, > __name__, __qualname__ > > __name__ and __qualname__ are obviously a good idea. The others I'm > not so sure about -- I'd worry that anyone who's reaching that deep > into the internals is not going to be satisfied by any half-way > attempts at compatibility, and it seems unlikely we can achieve > detailed compatibility at that level. But maybe it's not that bad; I > haven't looked at them in detail. Do you have any particular use cases > in mind for them? > I don't have a particular use case in mind, I'm just looking for completeness. They're right in the spec and I believe simple getters should do just fine (coroutines should have matching attributes which can be used directly). I'll send you a PR when I get around to it. > > -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomuxiong at gmx.com Tue Nov 29 12:39:45 2016 From: tomuxiong at gmx.com (Thomas Nyberg) Date: Tue, 29 Nov 2016 12:39:45 -0500 Subject: [Async-sig] A modest terminological proposal Message-ID: <4e5bb582-6daf-4759-f444-cc12d1218a51@gmx.com> This is in reply to this message in the archives: https://mail.python.org/pipermail/async-sig/2016-October/000141.html Nathaniel Smith: > I've found that when talking about async/await stuff recently, I've > mostly dropped the word "coroutine" from my vocabulary and replaced it > with "async function". I'm writing to suggest that we might want > to > make this switch as a community, and do it now, before the next 10x > increase in the async/await userbase. For what it's worth, I 100% agree. > *Accuracy*: Speaking of jargon, the term "coroutine" *is* an existing > piece of jargon in computer science, and our term and their term don't > quite match up. This isn't a huge deal, but it's unnecessary > confusion. According to Wikipedia I guess technically we don't even > have "true" coroutines, just "semicoroutines"? And every generator has > just as much claim to being a coroutine-in-the-CS-sense as an async > function does, but when we say coroutine we don't mean generators. > (Except when we do.) This confusion might partly reflect the somewhat > confusing transition from 'yield from' to async/await, as demonstrated > by the official doc's somewhat confusing definition of "coroutine": I've personally been very confused lately by python's terminology. I've always understood coroutines to have nothing a priori to do with event loops, scheduling. For example, I've understood coroutines as being objects that allow the following kind of flow (taken from https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines): var q := new queue coroutine produce loop while q is not full create some new items add the items to q yield to consume coroutine consume loop while q is not empty remove some items from q use the items yield to produce Python has basically supported this flow since 2.x (apart from the terminology issue of semi-coroutines mentioned above which I feel is relatively minor). This big point though is that being scheduled or asyncronous is kind of a separate property. Posix threads are implemented by feeding it a routine to start things off just like much of python's asyncronous stuff is built around coroutines, but I don't feel like the term "coroutines" should be merged into both just as routines aren't automatically tied to threads. Personally I think if anything the best possible comparison else where in computer science are the haskell thunks which delay execution until needed. That's the first thing I think about when I see how curio is setup for example. I recognize that it is very unlikely that this will change (hey language meaning changes over time), but I felt like I should reply after finding this thread because I personally have been very confused by the terminology used by python. (Personally I've been using coroutines by my definition for years using yield/send.) Thanks a lot for the write-up though. You saved my sanity a little bit... Cheers, Thomas From yarkot1 at gmail.com Wed Nov 30 18:21:06 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Wed, 30 Nov 2016 17:21:06 -0600 Subject: [Async-sig] A modest terminological proposal In-Reply-To: <4e5bb582-6daf-4759-f444-cc12d1218a51@gmx.com> References: <4e5bb582-6daf-4759-f444-cc12d1218a51@gmx.com> Message-ID: On Tue, Nov 29, 2016 at 11:39 AM, Thomas Nyberg wrote: > This is in reply to this message in the archives: > > https://mail.python.org/pipermail/async-sig/2016-October/000141.html > > Nathaniel Smith: > > I've found that when talking about async/await stuff recently, I've > > mostly dropped the word "coroutine" from my vocabulary and replaced it > > with "async function". I'm writing to suggest that we might want > to > > make this switch as a community, and do it now, before the next 10x > > increase in the async/await userbase. > > For what it's worth, I 100% agree. > ...yes. > > > *Accuracy*: Speaking of jargon, the term "coroutine" *is* an existing > > piece of jargon in computer science, and our term and their term don't > > quite match up. This isn't a huge deal, but it's unnecessary > > confusion. According to Wikipedia I guess technically we don't even > > have "true" coroutines, just "semicoroutines"? And every generator has > > just as much claim to being a coroutine-in-the-CS-sense as an async > > function does, but when we say coroutine we don't mean generators. > > (Except when we do.) This confusion might partly reflect the somewhat > > confusing transition from 'yield from' to async/await, as demonstrated > > by the official doc's somewhat confusing definition of "coroutine": > > I've personally been very confused lately by python's terminology. I've > always understood coroutines to have nothing a priori to do with event > loops, scheduling. For example, I've understood coroutines as being > objects that allow the following kind of flow (taken from > https://en.wikipedia.org/wiki/Coroutine#Comparison_with_subroutines): > I actually think this _is_ a bit of a big deal. Looking at things like: http://wla.berkeley.edu/~cs61a/fa11/lectures/streams.html#coroutines and (where generators were inspired from, for python) the Icon Programming Language book (e.g. co-expressions, and references to coroutines, e.g. p.119) http://www.cs.arizona.edu/icon/books.htm I think - really - async != coroutines, and (as others have said) muddies the waters, and kills some clarity. - Yarko > var q := new queue > > coroutine produce > loop > while q is not full > create some new items > add the items to q > yield to consume > > coroutine consume > loop > while q is not empty > remove some items from q > use the items > yield to produce > > Python has basically supported this flow since 2.x (apart from the > terminology issue of semi-coroutines mentioned above which I feel is > relatively minor). > > This big point though is that being scheduled or asyncronous is kind of > a separate property. Posix threads are implemented by feeding it a > routine to start things off just like much of python's asyncronous stuff > is built around coroutines, but I don't feel like the term "coroutines" > should be merged into both just as routines aren't automatically tied to > threads. > > Personally I think if anything the best possible comparison else where > in computer science are the haskell thunks which delay execution until > needed. That's the first thing I think about when I see how curio is > setup for example. > > I recognize that it is very unlikely that this will change (hey language > meaning changes over time), but I felt like I should reply after finding > this thread because I personally have been very confused by the > terminology used by python. (Personally I've been using coroutines by my > definition for years using yield/send.) > > Thanks a lot for the write-up though. You saved my sanity a little > bit... > > Cheers, > Thomas > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: