From yselivanov at gmail.com Tue Aug 2 10:15:47 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Tue, 2 Aug 2016 10:15:47 -0400 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: > To keep it simple, try thinking like this (and yes, Yury, apologies - this is now a side discussion, and not about this pep): everything in CPython is async, and if you don't want async, you don't need to know about, you run a single async task and don't need to know more... > > Can we get there? > That would be cool... So, essentially, something similar to Golang? I don?t know if that?s possible. It would require a complete CPython IO layer rewrite, integrating an event loop directly into the core, etc. The closest thing to that is gevent ? no async/await and all IO is non-blocking, but it has its own warts. Yury From yarkot1 at gmail.com Tue Aug 2 13:48:08 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Tue, 2 Aug 2016 12:48:08 -0500 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: On Tuesday, August 2, 2016, Yury Selivanov wrote: > > > To keep it simple, try thinking like this (and yes, Yury, apologies - > this is now a side discussion, and not about this pep): everything in > CPython is async, and if you don't want async, you don't need to know > about, you run a single async task and don't need to know more... > > > > Can we get there? > > That would be cool... > > > So, essentially, something similar to Golang? I don?t know if that?s > possible. It would require a complete CPython IO layer rewrite, > integrating an event loop directly into the core, etc. The closest thing > to that is gevent ? no async/await and all IO is non-blocking, but it has > its own warts. Warts in "all in on async", or on gevent (which isn't integrated. Very interested to hear your thoughts - redirect discussion to wherever is the right channel. - Yarko > > Yury -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Tue Aug 2 14:39:11 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Tue, 02 Aug 2016 18:39:11 +0000 Subject: [Async-sig] Warts both in asyncio way and greenlet-based approach Message-ID: Long story short: in async world every IO library should use asyncio. Unfortunately for the most popular libraries (requests, django, flask, sqlalchemy etc.) it's impossible to rewrite the code keeping backward compatibility. Much easier to create a new library than rewrite existing one. Regarding to gevent -- please read excellent Glyph's article 'Unyielding': https://glyph.twistedmatrix.com/2014/02/unyielding.html It covers pretty well why explicit yield points are better than implicit ones. Shortly explicit `await`'s give you 'atomic consistency' but using gevent you should expect context switch in every opcode instruction. It's much easier to test cover code with several (ok, many) context switch points. But test covering the code which may switch on every line and several times in the same line is practically impossible. Also my 2 cents. We used gevent. Even in very tiny application (command line client to upload huge files over HTTP, about 1k lines of code) under high load we got reports about gevent hub crashes. Even gevent core (hub is something like asyncio loop) is not 100% stable. Unfortunately we was not able to reproduce the problem by our test suite -- it requires really high load and occurs very rare. Well, the most gevent users can live with that -- gunicorn just restarts dead worker and that's it. But it is the sign for the problem: gevent crashes are extremely hard to debug. P.S. You might keep my writings about gevent as my private opinion, and it is. But from my perspective asyncio based solutions have much more predictable behavior and much more friendly to debug-and-fix problems. -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Wed Aug 3 10:27:03 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 3 Aug 2016 15:27:03 +0100 Subject: [Async-sig] Warts both in asyncio way and greenlet-based approach In-Reply-To: References: Message-ID: <841601D2-047E-4DDA-88CC-CBE853927BC7@lukasa.co.uk> > On 2 Aug 2016, at 19:39, Andrew Svetlov wrote: > > Long story short: in async world every IO library should use asyncio. > Unfortunately for the most popular libraries (requests, django, flask, sqlalchemy etc.) it's impossible to rewrite the code keeping backward compatibility. Much easier to create a new library than rewrite existing one. Removing my hyper hat for a moment and replacing it with my Requests hat: we disagree. Some of the discussion has happened on GitHub at [1], but I can summarise it again here. One of the nice things with an event loop is that you can retrofit synchronous code on top of it. That is, if you have an event loop, you can always turn your evented code (code that returned a Deferred (or Deferred-alike) or a coroutine) into synchronous code by simply running the event loop in the synchronous call. That is, given a method signature like this: async def get(url): return await do_the_work(url) It can be turned into this: def sync_get(url): return some_event_loop.run(get(url)) The reason Requests has not previously done this is that it required us to do one of two things. In the first instance, we could bundle or depend on an event loop (e.g. Twisted?s reactors). That?s annoying: it?s a lot of code that we don?t really care about in order to achieve a task that is fundamentally an implementation detail. The other option is to write backends for all possible event loops *and* a kind of event-loop-alike synchronous approach that can be run as a series of coroutines, but that is actually entirely synchronous under the covers (essentially, ship a coroutine runner that never waits for anything). Both of these were a lot of work for fairly minimal gain. However, with asyncio?s event loop becoming shipped with the language, and with the other event loop implementations shimming in compatibility layers, libraries like Requests have a perfect escape hatch. Now we only need *two* backends: one for the asyncio event loop, and one synchronous one. And if we?re prepared to depend on Python 3.4+, we only need *one* backend: asyncio, because we can rely on it being present. This drastic reduction in the space of event loops we need to support suddenly makes it much more viable to consider adjusting the way we do I/O. There?s still a *lot* of work there, and no-one has actually started sitting down to do the hard stuff, but it?s certainly not impossible any longer. Cory [1]: https://github.com/kennethreitz/requests/issues/1390#issuecomment-224772923 -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew.svetlov at gmail.com Thu Aug 4 19:47:42 2016 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Thu, 04 Aug 2016 23:47:42 +0000 Subject: [Async-sig] Warts both in asyncio way and greenlet-based approach In-Reply-To: <841601D2-047E-4DDA-88CC-CBE853927BC7@lukasa.co.uk> References: <841601D2-047E-4DDA-88CC-CBE853927BC7@lukasa.co.uk> Message-ID: @cory I didn't want to say "re-implementing requests in asyncio way is impossible". I just want to say: it's hard. Requires massive code rewriting. Moreover, it's pain to support both Python 2 and 3 with asyncio inthe same code base. Really only after dropping python 2 support you are able to adopt requests for asyncio. Hopefully after 2020. Requests library (as well as sqlalchemy) are very crucial scaffolds for python community, I suspect you will be forced to support Python 2 up to the official death of branch at least. On Wed, Aug 3, 2016 at 5:27 PM Cory Benfield wrote: > > On 2 Aug 2016, at 19:39, Andrew Svetlov wrote: > > Long story short: in async world every IO library should use asyncio. > Unfortunately for the most popular libraries (requests, django, flask, > sqlalchemy etc.) it's impossible to rewrite the code keeping backward > compatibility. Much easier to create a new library than rewrite existing > one. > > > Removing my hyper hat for a moment and replacing it with my Requests hat: > we disagree. Some of the discussion has happened on GitHub at [1], but I > can summarise it again here. > > One of the nice things with an event loop is that you can retrofit > synchronous code on top of it. That is, if you have an event loop, you can > always turn your evented code (code that returned a Deferred (or > Deferred-alike) or a coroutine) into synchronous code by simply running the > event loop in the synchronous call. That is, given a method signature like > this: > > async def get(url): > return await do_the_work(url) > > It can be turned into this: > > def sync_get(url): > return some_event_loop.run(get(url)) > > The reason Requests has not previously done this is that it required us to > do one of two things. In the first instance, we could bundle or depend on > an event loop (e.g. Twisted?s reactors). That?s annoying: it?s a lot of > code that we don?t really care about in order to achieve a task that is > fundamentally an implementation detail. The other option is to write > backends for all possible event loops *and* a kind of event-loop-alike > synchronous approach that can be run as a series of coroutines, but that is > actually entirely synchronous under the covers (essentially, ship a > coroutine runner that never waits for anything). Both of these were a lot > of work for fairly minimal gain. > > However, with asyncio?s event loop becoming shipped with the language, and > with the other event loop implementations shimming in compatibility layers, > libraries like Requests have a perfect escape hatch. Now we only need *two* > backends: one for the asyncio event loop, and one synchronous one. And if > we?re prepared to depend on Python 3.4+, we only need *one* backend: > asyncio, because we can rely on it being present. > > This drastic reduction in the space of event loops we need to support > suddenly makes it much more viable to consider adjusting the way we do I/O. > There?s still a *lot* of work there, and no-one has actually started > sitting down to do the hard stuff, but it?s certainly not impossible any > longer. > > Cory > > [1]: > https://github.com/kennethreitz/requests/issues/1390#issuecomment-224772923 > > -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Thu Aug 4 21:21:45 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Thu, 4 Aug 2016 18:21:45 -0700 Subject: [Async-sig] Warts both in asyncio way and greenlet-based approach In-Reply-To: References: <841601D2-047E-4DDA-88CC-CBE853927BC7@lukasa.co.uk> Message-ID: On Aug 4, 2016, at 4:47 PM, Andrew Svetlov wrote: > > @cory I didn't want to say "re-implementing requests in asyncio way is impossible". > > I just want to say: it's hard. I can't comment on the total difficulty as I'm not a requests maintainer, but I think you're overestimating it. > Requires massive code rewriting. In fact, requests had an async mode in the past, and if you look at the internal factoring, there are a totally manageable number of places where it actually depends on blocking. It would not be a "massive" rewrite, just a refactoring around those integration points between layers. > Moreover, it's pain to support both Python 2 and 3 with asyncio inthe same code base. Not really. You can't use the asyncio task scheduler, but we've been upgrading to be able to support asyncio from Twisted and it's a lot easier than many other Python 3 porting tasks :). > Really only after dropping python 2 support you are able to adopt requests for asyncio. Also not correct; we (Twisted) have supported many python3 features (for example; 'return' out of @inlineCallbacks-decorated generators) for several years despite being on python 2; nothing about the _interfaces_ that you have to support in order for asyncio users to use it are in any way version-specific. > Hopefully after 2020. We should be so lucky :). > Requests library (as well as sqlalchemy) are very crucial scaffolds for python community, I suspect you will be forced to support Python 2 up to the official death of branch at least. I don't think that's in question. -g > On Wed, Aug 3, 2016 at 5:27 PM Cory Benfield > wrote: > >> On 2 Aug 2016, at 19:39, Andrew Svetlov > wrote: >> >> Long story short: in async world every IO library should use asyncio. >> Unfortunately for the most popular libraries (requests, django, flask, sqlalchemy etc.) it's impossible to rewrite the code keeping backward compatibility. Much easier to create a new library than rewrite existing one. > > Removing my hyper hat for a moment and replacing it with my Requests hat: we disagree. Some of the discussion has happened on GitHub at [1], but I can summarise it again here. > > One of the nice things with an event loop is that you can retrofit synchronous code on top of it. That is, if you have an event loop, you can always turn your evented code (code that returned a Deferred (or Deferred-alike) or a coroutine) into synchronous code by simply running the event loop in the synchronous call. That is, given a method signature like this: > > async def get(url): > return await do_the_work(url) > > It can be turned into this: > > def sync_get(url): > return some_event_loop.run(get(url)) > > The reason Requests has not previously done this is that it required us to do one of two things. In the first instance, we could bundle or depend on an event loop (e.g. Twisted?s reactors). That?s annoying: it?s a lot of code that we don?t really care about in order to achieve a task that is fundamentally an implementation detail. The other option is to write backends for all possible event loops *and* a kind of event-loop-alike synchronous approach that can be run as a series of coroutines, but that is actually entirely synchronous under the covers (essentially, ship a coroutine runner that never waits for anything). Both of these were a lot of work for fairly minimal gain. > > However, with asyncio?s event loop becoming shipped with the language, and with the other event loop implementations shimming in compatibility layers, libraries like Requests have a perfect escape hatch. Now we only need *two* backends: one for the asyncio event loop, and one synchronous one. And if we?re prepared to depend on Python 3.4+, we only need *one* backend: asyncio, because we can rely on it being present. > > This drastic reduction in the space of event loops we need to support suddenly makes it much more viable to consider adjusting the way we do I/O. There?s still a *lot* of work there, and no-one has actually started sitting down to do the hard stuff, but it?s certainly not impossible any longer. > > Cory > > [1]: https://github.com/kennethreitz/requests/issues/1390#issuecomment-224772923 > -- > Thanks, > Andrew Svetlov > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Fri Aug 5 03:16:37 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 5 Aug 2016 08:16:37 +0100 Subject: [Async-sig] Warts both in asyncio way and greenlet-based approach In-Reply-To: References: <841601D2-047E-4DDA-88CC-CBE853927BC7@lukasa.co.uk> Message-ID: <7B284496-CF64-4E58-8B24-A416E299B0D8@lukasa.co.uk> > On 5 Aug 2016, at 02:21, Glyph Lefkowitz wrote: > > Not really. You can't use the asyncio task scheduler, but we've been upgrading to be able to support asyncio from Twisted and it's a lot easier than many other Python 3 porting tasks :). To follow up on Glyph?s point, what?s key is that the asyncio support doesn?t have to be present. It only has to be present if you assume that, in my previous example, I was always doing ?import asyncio; asyncio.run_until_complete? (not real, I need an event loop, but I?m not trying to dirty up the example here). However, run() can be *anything*. In particular, if we had ?separate but equal? backends, one for sync code and one for asyncio?s event loop, it doesn?t matter if the asyncio event loop isn?t present in the Python runtime: we just see the import error and say ?ok, only synchronous mode is available?. The goal there is to provide a transition: synchronous mode becomes the ?legacy? mode in Requests, with the expectation that we?ll transition over to a purely evented backend in some future release. >> Requests library (as well as sqlalchemy) are very crucial scaffolds for python community, I suspect you will be forced to support Python 2 up to the official death of branch at least. > > I don't think that's in question. I think we should support Python 2 up to end of support. However, I don?t think we should support Python 2 for *one day longer* than that unless a third-party vendor wants to cover the engineering cost of doing so. Additionally, I have exactly no problem with calling the synchronous legacy mode in Requests exactly that: it will remain unchanged and so will probably continue to work, but we are no longer developing it. Requests is an extremely important part of the community, which is why I have no problem using it to exert subtle but real pressure to encourage people to move to Python 3. Not right now, of course: we?ve still got a few years before that?s going to be necessary. =) Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmludo at gmail.com Sun Aug 7 17:44:55 2016 From: gmludo at gmail.com (Ludovic Gasc) Date: Sun, 7 Aug 2016 23:44:55 +0200 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: +1 for PEP, nothing more to add from technical point of view. An extra step to the right direction, at least to me. Thank you Yury for that :-) About side conversation on sync/async split world, except to force coroutines pattern usage like in Go, I don't see how we can become more implicit. Even if the zen of Python recommands to prefer an explicit approach, I see more explicit/implicit as a balance you must adjust between Simplicity/Flexibility than a binary choice. To me, the success of Python as language is also because you have a good balance between theses approaches, and the last move from "yield from" to "await" illustrates that: Hide the internal mechanisms of implementation, but keep the explicit way to declare that. Like Andrew Svetlov, I don't believe a lot in the implicit approach of Gevent, because very quickly, you need to understand the extra tools, like synchronization primitives. The fact to know if you need to prefix with "await" or not the call of the functions is the tree that hides the forest. With the async pattern, it's impossible to hide everything and everything will work automagically: You must understand a little bit what's happening, or it will be very complicated to debug. To me, you can hide everything only if you are really sure it will work 100% of time without human intervention, like with autonomous Google cars. However, it might be interesting to have an async "linter", that should list all blocking I/O code in async coroutines, to help new comers to find this type of bugs. But with the dynamic nature of Python, I don't know if it's realistic to try to implement that. To me, it should be a better answer than to try to remove all sync/async code differences. Moreover, I see the need of async libs as an extra opportunity to challenge and simplify the Python toolbox. For now, with aiohttp, you have an unified API for HTTP in general, contrary in sync world with requests and flask for example. At least to me, a client and a server are only two sides of the same piece. More true with p2p protocols. As discussed several times, the next level might be more code reuse like suggested by David Beazley with SansIO, split protocol and I/O handling: https://twitter.com/dabeaz/status/761599925444550656?lang=fr https://github.com/brettcannon/sans-io I don't know yet if the benefit to share more code between implementations will be more important than the potential complexity code increase. The only point I'm sure for now: I'm preparing the pop-corn to watch the next episodes: curious to see what are the next ideas/implementations will emerge ;-) At least to me, it's more interesting than follow a TV serie, thank you for that :-) Have a nice week. Ludovic Gasc (GMLudo) http://www.gmludo.eu/ On 29 Jul 2016 20:50, "Yarko Tymciurak" wrote: > > > On Friday, July 29, 2016, Yury Selivanov wrote: > >> Comments inlined: >> >> >> > On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak wrote: >> > >> > Hmm... I think we need to think about a future where, >> programmatically, there's little-to no distinction between async and >> synchronous functions. Pushing this down deeper in the system is the way to >> go. For one, it will serve simple multi-core use once gilectomy is >> completed (it, or something effectively equivalent will complete). For >> another, this is the path to reducing the functionally "useless" rewrite >> efforts of libraries (e.g. github.com/aio-libs), which somehow resemble >> all the efforts of migrating libraries from 2-3 (loosely). The resistance >> and unexpected time that 2-3 migration experienced won't readily be >> mimicked in async tasks - too much effort to get computer and I/O bound >> benefits? Maintain two versions of needed libraries, or jump languages is >> what will increasingly happen in the distributed (and more so IOT) world. >> >> When and *if* gilectomy is completed (or another project to remove the >> GIL), we will be able to do this: >> >> 1) Run existing async/await applications as is, but instead of running a >> process per core, we will be able to run a single process with many >> threads. Likely one asyncio (or other) event loop per thread. This is >> very speculative, but possible in theory. >> >> 2) Run existing blocking IO applications in several threads in one >> process. This is something that only sounds like an easy thing to do, I >> suspect that a lot of code will break (or dead lock) when the GIL is >> removed. Even if everything works perfectly well, threads aren?t answer to >> all problems ? try to manage 1000s of them. >> >> Long story short, even if we had no GIL at all, having async/await (and >> non-blocking IO) would make sense. And if you have async/await, with GIL >> or without, you will inevitably have different APIs and different IO >> low-level libs that drive them. >> >> There are ways to lessen the pain. For instance, I like Cory?s approach >> with hyper - implement protocols separately from IO, so that it?s easy to >> port them to various sync and async frameworks. >> >> > >> > Time to think about paving the way to async-as first class citizen >> world. >> > >> > That's probably too much for this PEP, but the topic (a- prefixing) is >> a good canary for the bigger picture we need to start mulling over. >> > >> > So in this context (and in general w/ async) asking the question "can >> we make it so it doesn't matter?" is a good one to always be asking - it >> will get is there. >> > >> >> Unfortunately there is no way to use the same APIs for both async/await >> and synchronous world. At least for CPython builtin types they have to >> have different names. >> >> I?m fine to discuss the ?a? prefix, but I?m a bit afraid that focusing on >> it too much will distract us from the PEP and details of it that really >> matter. >> >> Yury > > > To keep it simple, try thinking like this (and yes, Yury, apologies - this > is now a side discussion, and not about this pep): everything in CPython > is async, and if you don't want async, you don't need to know about, you > run a single async task and don't need to know more... > > Can we get there? > That would be cool... > > - Yarko > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yarkot1 at gmail.com Mon Aug 8 02:34:22 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Mon, 8 Aug 2016 01:34:22 -0500 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: I still have to wonder, though, how an async repl, from the inside-out,which handles a single task by default (synchronous equivalent) would be anything less than explicit, or would complicate much (if anything - I suspect a significant amount of the opposite). Regardless, I am grateful for the discussions. - Yarko On Sunday, August 7, 2016, Ludovic Gasc wrote: > +1 for PEP, nothing more to add from technical point of view. > An extra step to the right direction, at least to me. > Thank you Yury for that :-) > > About side conversation on sync/async split world, except to force > coroutines pattern usage like in Go, I don't see how we can become more > implicit. > Even if the zen of Python recommands to prefer an explicit approach, I see > more explicit/implicit as a balance you must adjust between > Simplicity/Flexibility than a binary choice. > > To me, the success of Python as language is also because you have a > good balance between theses approaches, and the last move from "yield from" > to "await" illustrates that: Hide the internal mechanisms of > implementation, but keep the explicit way to declare that. > > Like Andrew Svetlov, I don't believe a lot in the implicit approach of > Gevent, because very quickly, you need to understand the extra tools, like > synchronization primitives. The fact to know if you need to prefix with > "await" or not the call of the functions is the tree that hides the forest. > > With the async pattern, it's impossible to hide everything and everything > will work automagically: You must understand a little bit what's happening, > or it will be very complicated to debug. > > To me, you can hide everything only if you are really sure it will work > 100% of time without human intervention, like with autonomous Google cars. > > However, it might be interesting to have an async "linter", that should > list all blocking I/O code in async coroutines, to help new comers to find > this type of bugs. > But with the dynamic nature of Python, I don't know if it's realistic to > try to implement that. > To me, it should be a better answer than to try to remove all sync/async > code differences. > > Moreover, I see the need of async libs as an extra opportunity to > challenge and simplify the Python toolbox. > > For now, with aiohttp, you have an unified API for HTTP in general, > contrary in sync world with requests and flask for example. > At least to me, a client and a server are only two sides of the same > piece. More true with p2p protocols. > > As discussed several times, the next level might be more code reuse like > suggested by David Beazley with SansIO, split protocol and I/O handling: > https://twitter.com/dabeaz/status/761599925444550656?lang=fr > > https://github.com/brettcannon/sans-io > > I don't know yet if the benefit to share more code between implementations > will be more important than the potential complexity code increase. > > The only point I'm sure for now: I'm preparing the pop-corn to watch the > next episodes: curious to see what are the next ideas/implementations will > emerge ;-) > At least to me, it's more interesting than follow a TV serie, thank you > for that :-) > > Have a nice week. > > Ludovic Gasc (GMLudo) > http://www.gmludo.eu/ > > On 29 Jul 2016 20:50, "Yarko Tymciurak" > wrote: > >> >> >> On Friday, July 29, 2016, Yury Selivanov > > wrote: >> >>> Comments inlined: >>> >>> >>> > On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak >>> wrote: >>> > >>> > Hmm... I think we need to think about a future where, >>> programmatically, there's little-to no distinction between async and >>> synchronous functions. Pushing this down deeper in the system is the way to >>> go. For one, it will serve simple multi-core use once gilectomy is >>> completed (it, or something effectively equivalent will complete). For >>> another, this is the path to reducing the functionally "useless" rewrite >>> efforts of libraries (e.g. github.com/aio-libs), which somehow resemble >>> all the efforts of migrating libraries from 2-3 (loosely). The resistance >>> and unexpected time that 2-3 migration experienced won't readily be >>> mimicked in async tasks - too much effort to get computer and I/O bound >>> benefits? Maintain two versions of needed libraries, or jump languages is >>> what will increasingly happen in the distributed (and more so IOT) world. >>> >>> When and *if* gilectomy is completed (or another project to remove the >>> GIL), we will be able to do this: >>> >>> 1) Run existing async/await applications as is, but instead of running a >>> process per core, we will be able to run a single process with many >>> threads. Likely one asyncio (or other) event loop per thread. This is >>> very speculative, but possible in theory. >>> >>> 2) Run existing blocking IO applications in several threads in one >>> process. This is something that only sounds like an easy thing to do, I >>> suspect that a lot of code will break (or dead lock) when the GIL is >>> removed. Even if everything works perfectly well, threads aren?t answer to >>> all problems ? try to manage 1000s of them. >>> >>> Long story short, even if we had no GIL at all, having async/await (and >>> non-blocking IO) would make sense. And if you have async/await, with GIL >>> or without, you will inevitably have different APIs and different IO >>> low-level libs that drive them. >>> >>> There are ways to lessen the pain. For instance, I like Cory?s approach >>> with hyper - implement protocols separately from IO, so that it?s easy to >>> port them to various sync and async frameworks. >>> >>> > >>> > Time to think about paving the way to async-as first class citizen >>> world. >>> > >>> > That's probably too much for this PEP, but the topic (a- prefixing) is >>> a good canary for the bigger picture we need to start mulling over. >>> > >>> > So in this context (and in general w/ async) asking the question "can >>> we make it so it doesn't matter?" is a good one to always be asking - it >>> will get is there. >>> > >>> >>> Unfortunately there is no way to use the same APIs for both async/await >>> and synchronous world. At least for CPython builtin types they have to >>> have different names. >>> >>> I?m fine to discuss the ?a? prefix, but I?m a bit afraid that focusing >>> on it too much will distract us from the PEP and details of it that really >>> matter. >>> >>> Yury >> >> >> To keep it simple, try thinking like this (and yes, Yury, apologies - >> this is now a side discussion, and not about this pep): everything in >> CPython is async, and if you don't want async, you don't need to know >> about, you run a single async task and don't need to know more... >> >> Can we get there? >> That would be cool... >> >> - Yarko >> >> _______________________________________________ >> Async-sig mailing list >> Async-sig at python.org >> >> https://mail.python.org/mailman/listinfo/async-sig >> Code of Conduct: https://www.python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yarkot1 at gmail.com Mon Aug 8 02:56:56 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Mon, 8 Aug 2016 01:56:56 -0500 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: On Monday, August 8, 2016, Yarko Tymciurak wrote: > I still have to wonder, though, how an async repl, from the > inside-out,which handles a single task by default (synchronous equivalent) > would be anything less than explicit, or would complicate much (if anything > - I suspect a significant amount of the opposite). > > Regardless, I am grateful for the discussions. > > - Yarko > > On Sunday, August 7, 2016, Ludovic Gasc > wrote: > >> +1 for PEP, nothing more to add from technical point of view. >> An extra step to the right direction, at least to me. >> Thank you Yury for that :-) >> >> About side conversation on sync/async split world, except to force >> coroutines pattern usage like in Go, I don't see how we can become more >> implicit. >> Even if the zen of Python recommands to prefer an explicit approach, I >> see more explicit/implicit as a balance you must adjust between >> Simplicity/Flexibility than a binary choice. >> >> To me, the success of Python as language is also because you have a >> good balance between theses approaches, and the last move from "yield from" >> to "await" illustrates that: Hide the internal mechanisms of >> implementation, but keep the explicit way to declare that. >> >> Like Andrew Svetlov, I don't believe a lot in the implicit approach of >> Gevent, because very quickly, you need to understand the extra tools, like >> synchronization primitives. The fact to know if you need to prefix with >> "await" or not the call of the functions is the tree that hides the forest. >> >> With the async pattern, it's impossible to hide everything and everything >> will work automagically: You must understand a little bit what's happening, >> or it will be very complicated to debug. >> >> To me, you can hide everything only if you are really sure it will work >> 100% of time without human intervention, like with autonomous Google cars. >> >> However, it might be interesting to have an async "linter", that should >> list all blocking I/O code in async coroutines, to help new comers to find >> this type of bugs. >> But with the dynamic nature of Python, I don't know if it's realistic to >> try to implement that. >> To me, it should be a better answer than to try to remove all sync/async >> code differences. >> >> Moreover, I see the need of async libs as an extra opportunity to >> challenge and simplify the Python toolbox. >> >> For now, with aiohttp, you have an unified API for HTTP in general, >> contrary in sync world with requests and flask for example. >> At least to me, a client and a server are only two sides of the same >> piece. More true with p2p protocols. >> >> As discussed several times, the next level might be more code reuse like >> suggested by David Beazley with SansIO, split protocol and I/O handling: >> https://twitter.com/dabeaz/status/761599925444550656?lang=fr >> >> https://github.com/brettcannon/sans-io >> > Question: isn't SansIO / Corey's work just a specific instance of Bob Martin's "Clean Architecture"? It sounds familiar to me, when thinking of Brandon Rhode's 2014 PyOhio talk, and his recast of the topic in Warsaw in 2015. It seems like it... If so, then perhaps async aspects are just a second aspect. I don't know yet if the benefit to share more code between implementations >> will be more important than the potential complexity code increase. >> >> The only point I'm sure for now: I'm preparing the pop-corn to watch the >> next episodes: curious to see what are the next ideas/implementations will >> emerge ;-) >> At least to me, it's more interesting than follow a TV serie, thank you >> for that :-) >> >> Have a nice week. >> >> Ludovic Gasc (GMLudo) >> http://www.gmludo.eu/ >> >> On 29 Jul 2016 20:50, "Yarko Tymciurak" wrote: >> >>> >>> >>> On Friday, July 29, 2016, Yury Selivanov wrote: >>> >>>> Comments inlined: >>>> >>>> >>>> > On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak >>>> wrote: >>>> > >>>> > Hmm... I think we need to think about a future where, >>>> programmatically, there's little-to no distinction between async and >>>> synchronous functions. Pushing this down deeper in the system is the way to >>>> go. For one, it will serve simple multi-core use once gilectomy is >>>> completed (it, or something effectively equivalent will complete). For >>>> another, this is the path to reducing the functionally "useless" rewrite >>>> efforts of libraries (e.g. github.com/aio-libs), which somehow >>>> resemble all the efforts of migrating libraries from 2-3 (loosely). The >>>> resistance and unexpected time that 2-3 migration experienced won't readily >>>> be mimicked in async tasks - too much effort to get computer and I/O bound >>>> benefits? Maintain two versions of needed libraries, or jump languages is >>>> what will increasingly happen in the distributed (and more so IOT) world. >>>> >>>> When and *if* gilectomy is completed (or another project to remove the >>>> GIL), we will be able to do this: >>>> >>>> 1) Run existing async/await applications as is, but instead of running >>>> a process per core, we will be able to run a single process with many >>>> threads. Likely one asyncio (or other) event loop per thread. This is >>>> very speculative, but possible in theory. >>>> >>>> 2) Run existing blocking IO applications in several threads in one >>>> process. This is something that only sounds like an easy thing to do, I >>>> suspect that a lot of code will break (or dead lock) when the GIL is >>>> removed. Even if everything works perfectly well, threads aren?t answer to >>>> all problems ? try to manage 1000s of them. >>>> >>>> Long story short, even if we had no GIL at all, having async/await (and >>>> non-blocking IO) would make sense. And if you have async/await, with GIL >>>> or without, you will inevitably have different APIs and different IO >>>> low-level libs that drive them. >>>> >>>> There are ways to lessen the pain. For instance, I like Cory?s >>>> approach with hyper - implement protocols separately from IO, so that it?s >>>> easy to port them to various sync and async frameworks. >>>> >>>> > >>>> > Time to think about paving the way to async-as first class citizen >>>> world. >>>> > >>>> > That's probably too much for this PEP, but the topic (a- prefixing) >>>> is a good canary for the bigger picture we need to start mulling over. >>>> > >>>> > So in this context (and in general w/ async) asking the question "can >>>> we make it so it doesn't matter?" is a good one to always be asking - it >>>> will get is there. >>>> > >>>> >>>> Unfortunately there is no way to use the same APIs for both async/await >>>> and synchronous world. At least for CPython builtin types they have to >>>> have different names. >>>> >>>> I?m fine to discuss the ?a? prefix, but I?m a bit afraid that focusing >>>> on it too much will distract us from the PEP and details of it that really >>>> matter. >>>> >>>> Yury >>> >>> >>> To keep it simple, try thinking like this (and yes, Yury, apologies - >>> this is now a side discussion, and not about this pep): everything in >>> CPython is async, and if you don't want async, you don't need to know >>> about, you run a single async task and don't need to know more... >>> >>> Can we get there? >>> That would be cool... >>> >>> - Yarko >>> >>> _______________________________________________ >>> Async-sig mailing list >>> Async-sig at python.org >>> https://mail.python.org/mailman/listinfo/async-sig >>> Code of Conduct: https://www.python.org/psf/codeofconduct/ >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Mon Aug 8 05:45:43 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 8 Aug 2016 10:45:43 +0100 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> > On 8 Aug 2016, at 07:56, Yarko Tymciurak wrote: > > Question: isn't SansIO / Corey's work just a specific instance of Bob Martin's "Clean Architecture"? It sounds familiar to me, when thinking of Brandon Rhode's 2014 PyOhio talk, and his recast of the topic in Warsaw in 2015. It seems like it... If so, then perhaps async aspects are just a second aspect. Yeah, so this is worth highlighting more clearly. =) I apologise to the list in advance for this, but this is going to be something of a digression, so I have re-titled the thread. I have invented and pioneered *nothing* here. Designing protocol implementations this way is not new in the world of software development. No-one is going to give me a medal or name a design pattern after me, and I don?t even think that pushing for this design pattern will help me move up my employer?s technical career path the way spending time on writing RFCs would have. However, I think we can safely say that the Python community has not effectively done this over our twenty-plus year lifetime. Brett?s ?sans-io? page is a beautiful example of this: currently we can find maybe *three* libraries that tick the box of being I/O-independent. This, in my mind, points to a cultural problem in the Python community. For whatever reason, we have tricked ourselves into abandoning what I would call ?good software design?. I have my theories for why this is. A huge factor of this is the relative ease-of-use of sockets and other I/O tools in Python (relative to C, that is: languages that also have this relative ease of use have similar problems, such as Javascript and Ruby). Another factor is the Python community?s love of beautiful APIs. Libraries with beautiful APIs provide such an appeal to programmers that there is huge incentive to build such a library. Building such a library requires including your I/O, and in the absence of any I/O-free implementations of a protocol you are almost certainly going to take the path of least resistance, which means intermingling your I/O and event loop primitives with your protocol code. And I?m not criticising programmers that do this. As I pointed out with the fake title of my talk, I am a programmer that does this! Almost all the best programmers our community has to offer have done this and continue to do this. It really is easier. And trust me, as someone who has written one: there is very little that is sexy about writing a protocol library that does no I/O. It doesn?t make you famous, and it doesn?t make anyone applaud your design sense. They?re extremely boring and prosaic, and they rarely solve the actual problem you have: they are a step on the road to solving the problem generally, but not specifically. So am I talking about the same thing as Brandon? Yes, yes I am. And Brandon is a better speaker than I am, so I agree that it?s weird that I?m getting more credit for this than he is. But I suspect that I?ve had some advantages that Brandon didn?t have. The first is that, by sheer good luck, I?ve managed to tap into a zeitgeist and be in the right time at the right place to deliver this message. Dave Beazley?s work on curio here is helping, because of curio?s sheer incompatibility with the other event loop approaches, which means that his work and mine have a nice symbiosis. Nathaniel and I have managed to give him the building blocks to demonstrate curio?s effectiveness without him needing to be an expert in HTTP. The second is that, I think, Brandon was targeting a different audience. Brandon was trying to talk to the general case, but he stopped short of the case I made. If you go back to watch Brandon?s talk, he talks about hoisting your I/O to the top of your control flow, but never leaps forward to say ?and this has a very specific application to any tool or library that talks a network protocol?. My argument is actually *more specific* than Brandon?s: I am saying that you can strip out the low-level stuff into an entirely separate logical codebase, and then not just change the *type* of I/O you do, but change the complete style in which you do it. Brandon?s argument isn?t so much about code reuse as it is about code maintenance. That said, I think the real reason people are talking about my work and not Brandon?s is because, while we both said ?You should hoist your I/O out of your logic?, I got to follow up with ?and I have done it and released the code and it?s actually being used today in projects you know about?. That second argument really does help, because whenever you tell people to totally change the way they?re writing something, they have a tendency to go ?Oh, yeah, but that doesn?t work for *my* project?. Being able to say ?look, I did it, it really does work, and here are all the ways that it let me build a better thing?. That said, I?d love to have Brandon weigh in on this too. I do feel a bit bad that I am re-treading ground that others have covered before, but?hell, we can?t all be top-tier programmers! If all I say is that I have repackaged an idea my intellectual betters have previously made such that it?s more appealing to the masses, I think I can call myself happy with that. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmludo at gmail.com Mon Aug 8 06:16:25 2016 From: gmludo at gmail.com (Ludovic Gasc) Date: Mon, 8 Aug 2016 12:16:25 +0200 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: 2016-08-08 8:56 GMT+02:00 Yarko Tymciurak : > > > On Monday, August 8, 2016, Yarko Tymciurak wrote: > >> I still have to wonder, though, how an async repl, from the >> inside-out,which handles a single task by default (synchronous equivalent) >> would be anything less than explicit, or would complicate much (if anything >> - I suspect a significant amount of the opposite). >> >> Regardless, I am grateful for the discussions. >> >> - Yarko >> >> On Sunday, August 7, 2016, Ludovic Gasc wrote: >> >>> +1 for PEP, nothing more to add from technical point of view. >>> An extra step to the right direction, at least to me. >>> Thank you Yury for that :-) >>> >>> About side conversation on sync/async split world, except to force >>> coroutines pattern usage like in Go, I don't see how we can become more >>> implicit. >>> Even if the zen of Python recommands to prefer an explicit approach, I >>> see more explicit/implicit as a balance you must adjust between >>> Simplicity/Flexibility than a binary choice. >>> >>> To me, the success of Python as language is also because you have a >>> good balance between theses approaches, and the last move from "yield from" >>> to "await" illustrates that: Hide the internal mechanisms of >>> implementation, but keep the explicit way to declare that. >>> >>> Like Andrew Svetlov, I don't believe a lot in the implicit approach of >>> Gevent, because very quickly, you need to understand the extra tools, like >>> synchronization primitives. The fact to know if you need to prefix with >>> "await" or not the call of the functions is the tree that hides the forest. >>> >>> With the async pattern, it's impossible to hide everything and >>> everything will work automagically: You must understand a little bit what's >>> happening, or it will be very complicated to debug. >>> >>> To me, you can hide everything only if you are really sure it will work >>> 100% of time without human intervention, like with autonomous Google cars. >>> >>> However, it might be interesting to have an async "linter", that should >>> list all blocking I/O code in async coroutines, to help new comers to find >>> this type of bugs. >>> But with the dynamic nature of Python, I don't know if it's realistic to >>> try to implement that. >>> To me, it should be a better answer than to try to remove all sync/async >>> code differences. >>> >>> Moreover, I see the need of async libs as an extra opportunity to >>> challenge and simplify the Python toolbox. >>> >>> For now, with aiohttp, you have an unified API for HTTP in general, >>> contrary in sync world with requests and flask for example. >>> At least to me, a client and a server are only two sides of the same >>> piece. More true with p2p protocols. >>> >>> As discussed several times, the next level might be more code reuse like >>> suggested by David Beazley with SansIO, split protocol and I/O handling: >>> https://twitter.com/dabeaz/status/761599925444550656?lang=fr >>> >>> https://github.com/brettcannon/sans-io >>> >> > Question: isn't SansIO / Corey's work just a specific instance of Bob > Martin's "Clean Architecture"? It sounds familiar to me, when thinking of > Brandon Rhode's 2014 PyOhio talk, and his recast of the topic in Warsaw in > 2015. It seems like it... If so, then perhaps async aspects are just a > second aspect. > Maybe that sync/async dichotomy might be a concrete use case to justify this split. Nevertheless, it shouldn't be the first case where, theoretically, it's better to split layers, and finally, it's counter-productive during the implementation. Only one method to know that: Try to code to see what's happen. Certainly some protocols/transports should be easier to have this split than others: Interesting to know if somebody has already tried to have QUIC and HTTP/2 in the same time with Python. > > I don't know yet if the benefit to share more code between implementations >>> will be more important than the potential complexity code increase. >>> >>> The only point I'm sure for now: I'm preparing the pop-corn to watch the >>> next episodes: curious to see what are the next ideas/implementations will >>> emerge ;-) >>> At least to me, it's more interesting than follow a TV serie, thank you >>> for that :-) >>> >>> Have a nice week. >>> >>> Ludovic Gasc (GMLudo) >>> http://www.gmludo.eu/ >>> >>> On 29 Jul 2016 20:50, "Yarko Tymciurak" wrote: >>> >>>> >>>> >>>> On Friday, July 29, 2016, Yury Selivanov wrote: >>>> >>>>> Comments inlined: >>>>> >>>>> >>>>> > On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak >>>>> wrote: >>>>> > >>>>> > Hmm... I think we need to think about a future where, >>>>> programmatically, there's little-to no distinction between async and >>>>> synchronous functions. Pushing this down deeper in the system is the way to >>>>> go. For one, it will serve simple multi-core use once gilectomy is >>>>> completed (it, or something effectively equivalent will complete). For >>>>> another, this is the path to reducing the functionally "useless" rewrite >>>>> efforts of libraries (e.g. github.com/aio-libs), which somehow >>>>> resemble all the efforts of migrating libraries from 2-3 (loosely). The >>>>> resistance and unexpected time that 2-3 migration experienced won't readily >>>>> be mimicked in async tasks - too much effort to get computer and I/O bound >>>>> benefits? Maintain two versions of needed libraries, or jump languages is >>>>> what will increasingly happen in the distributed (and more so IOT) world. >>>>> >>>>> When and *if* gilectomy is completed (or another project to remove the >>>>> GIL), we will be able to do this: >>>>> >>>>> 1) Run existing async/await applications as is, but instead of running >>>>> a process per core, we will be able to run a single process with many >>>>> threads. Likely one asyncio (or other) event loop per thread. This is >>>>> very speculative, but possible in theory. >>>>> >>>>> 2) Run existing blocking IO applications in several threads in one >>>>> process. This is something that only sounds like an easy thing to do, I >>>>> suspect that a lot of code will break (or dead lock) when the GIL is >>>>> removed. Even if everything works perfectly well, threads aren?t answer to >>>>> all problems ? try to manage 1000s of them. >>>>> >>>>> Long story short, even if we had no GIL at all, having async/await >>>>> (and non-blocking IO) would make sense. And if you have async/await, with >>>>> GIL or without, you will inevitably have different APIs and different IO >>>>> low-level libs that drive them. >>>>> >>>>> There are ways to lessen the pain. For instance, I like Cory?s >>>>> approach with hyper - implement protocols separately from IO, so that it?s >>>>> easy to port them to various sync and async frameworks. >>>>> >>>>> > >>>>> > Time to think about paving the way to async-as first class citizen >>>>> world. >>>>> > >>>>> > That's probably too much for this PEP, but the topic (a- prefixing) >>>>> is a good canary for the bigger picture we need to start mulling over. >>>>> > >>>>> > So in this context (and in general w/ async) asking the question >>>>> "can we make it so it doesn't matter?" is a good one to always be asking - >>>>> it will get is there. >>>>> > >>>>> >>>>> Unfortunately there is no way to use the same APIs for both >>>>> async/await and synchronous world. At least for CPython builtin types they >>>>> have to have different names. >>>>> >>>>> I?m fine to discuss the ?a? prefix, but I?m a bit afraid that focusing >>>>> on it too much will distract us from the PEP and details of it that really >>>>> matter. >>>>> >>>>> Yury >>>> >>>> >>>> To keep it simple, try thinking like this (and yes, Yury, apologies - >>>> this is now a side discussion, and not about this pep): everything in >>>> CPython is async, and if you don't want async, you don't need to know >>>> about, you run a single async task and don't need to know more... >>>> >>>> Can we get there? >>>> That would be cool... >>>> >>>> - Yarko >>>> >>>> _______________________________________________ >>>> Async-sig mailing list >>>> Async-sig at python.org >>>> https://mail.python.org/mailman/listinfo/async-sig >>>> Code of Conduct: https://www.python.org/psf/codeofconduct/ >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Aug 8 06:53:05 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 8 Aug 2016 03:53:05 -0700 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> Message-ID: > On Aug 8, 2016, at 2:45 AM, Cory Benfield wrote: > >> On 8 Aug 2016, at 07:56, Yarko Tymciurak > wrote: >> >> Question: isn't SansIO / Corey's work just a specific instance of Bob Martin's "Clean Architecture"? It sounds familiar to me, when thinking of Brandon Rhode's 2014 PyOhio talk, and his recast of the topic in Warsaw in 2015. It seems like it... If so, then perhaps async aspects are just a second aspect. > > > Yeah, so this is worth highlighting more clearly. =) I apologise to the list in advance for this, but this is going to be something of a digression, so I have re-titled the thread. > > I have invented and pioneered *nothing* here. Designing protocol implementations this way is not new in the world of software development. No-one is going to give me a medal or name a design pattern after me, and I don?t even think that pushing for this design pattern will help me move up my employer?s technical career path the way spending time on writing RFCs would have. I wouldn't be so sure about that - I have definitely been referring to the principle that one should design an I/O free layer in every system as "Benfield's Law" in recent conversations ;-). > That said, I?d love to have Brandon weigh in on this too. I do feel a bit bad that I am re-treading ground that others have covered before, but?hell, we can?t all be top-tier programmers! If all I say is that I have repackaged an idea my intellectual betters have previously made such that it?s more appealing to the masses, I think I can call myself happy with that. I really don't think you should feel even a bit bad here. I'm one of the people who has been trying to tell people to do this for the last decade or so, and frankly, I'd been expressing myself poorly. There's also a "do as I say, not as I do" problem here: I learned this lesson _after_ I had implemented a bunch of protocols the wrong way (stacks and stacks of inheritance, tight I/O coupling), and then decided I'd surely do the _next_ one right. But of course then I was buried under the crushing maintenance burden of the existing stuff and I still have not managed to crawl my way out. Certainly, none of the popular code I've written works this way. As a result, I feel tremendously grateful that you have managed to noticeably increase the popularity of this concept, and as far as I'm concerned, you deserve quite a bit of credit. Given the number of people publicly associated with it now, clearly it's something we all thought was important to advocate for (and did successfully, to the extent that some people have learned those lessons!), but nevertheless nobody that I'm aware of, especially in the Python community, has managed as impactful a presentation as you have. This is not entirely an accident of history - it was a very good talk that was deceptively simple-looking but belied a lot of deep expertise. So don't sell yourself short: you may not have invented the concept, but I know I could not have given that talk in quite that way; I'm not sure any of the giants whose shoulders you stand upon could have either. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Mon Aug 8 07:24:39 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 8 Aug 2016 12:24:39 +0100 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: > On 8 Aug 2016, at 11:16, Ludovic Gasc wrote: > > Certainly some protocols/transports should be easier to have this split than others: Interesting to know if somebody has already tried to have QUIC and HTTP/2 in the same time with Python. AFAIK they haven?t. This is partly because there?s no good QUIC implementation to bind from Python at this time. Chromium?s QUIC library requires a giant pool of custom C++ to bind it appropriately, and Go?s implementation includes a gigantic runtime and is quite large. As and when a good OSS QUIC library starts to surface, I?ll be able to answer this question more effectively. But I?m not expecting a huge issue. =) Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at dabeaz.com Mon Aug 8 08:26:34 2016 From: dave at dabeaz.com (David Beazley) Date: Mon, 8 Aug 2016 07:26:34 -0500 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> Message-ID: <380E3FF1-706F-419E-9532-8BAF42AA11BE@dabeaz.com> > On Aug 8, 2016, at 4:45 AM, Cory Benfield wrote: >> > > The first is that, by sheer good luck, I?ve managed to tap into a zeitgeist and be in the right time at the right place to deliver this message. Dave Beazley?s work on curio here is helping, because of curio?s sheer incompatibility with the other event loop approaches, which means that his work and mine have a nice symbiosis. Nathaniel and I have managed to give him the building blocks to demonstrate curio?s effectiveness without him needing to be an expert in HTTP. > Chiming in on the "zeitgeist" comment for a moment, I've wondered for a long time why Python can't reinvent itself in the area of I/O (and maybe systems programming generally). Honestly, I feel like a whole lot of time has been burned up thinking about Python 2/3 compatibility instead of looking forward with futuristic new projects and ideas. Perhaps "async/await" serves as a catalyst to rethink some of these things. A lot of my work with async/await is really focused on exploring the API space with it--well, at least seeing how much I can twist that part of the language in diabolical ways. The protocol issue is real though. Sure, I could probably bang out a passable HTTP/0.9 protocol in an afternoon, but try to tackle something modern like HTTP/2? No way. I'm totally out of my element with something like that. Having an I/O-free implementation of it is cool. It would be pretty neat to have something like that for various other things too (Redis, MySQL, postgres, ZeroMQ, etc.). Cheers, Dave From yarkot1 at gmail.com Mon Aug 8 10:38:08 2016 From: yarkot1 at gmail.com (Yarko Tymciurak) Date: Mon, 8 Aug 2016 09:38:08 -0500 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <380E3FF1-706F-419E-9532-8BAF42AA11BE@dabeaz.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <380E3FF1-706F-419E-9532-8BAF42AA11BE@dabeaz.com> Message-ID: Thanks Cory - I wasn't so much saying "Isn't Cory doing kinda what Brandon said" as I was saying "Isn't this kinda the more general stuff that Bob Martin talks (tries to talk about?) with Clean Architechture / Clean Coding (which Brandon - first I saw explicitly - shows a bit of examples of in the Python context in his PyOhio, and how that helps testability ... and I suppose maintainability). So, less than my intending to point-to / assign credit (credit is due _everywhere_, honestly great stuff abounds!), I was hoping to say "hey - we've just scratched the surface - we can take this more broadly!" For me, the interesting question (in looking at what coding errors I've seen which opened my eyes to all this, as entangled as they've been in several kinds all mixed in) - isn't async in general one of these elements with potential for "clean", for thinking about taking it further in python, in effect doing with the language sort of a similar thing which Cory has demonstrated w/ protocols? I don't know, but I am curios (pun not intended). Dave Beazley's curio (which I had the pleasure of saying "quit talking, show me running code!" early on - and then "wow!"), and other things are leading me to think our / Python community's perspective on async might (?) need a bigger shift, a bigger kick-out-of-current-ways-of-thinking. Yury's comment "you mean put async into the core, like go?" was probably spot on as far as understanding my thinking - but not to mimic, but rather to honestly explore the bigger library implications - maintenance and simplicity and performance - and the "keeping-you-out-of-trouble" effect that (hopefully) might exist for application coders. At this year's PyOhio, Dave mentioned a project moving from python to go because of this. I saw in enough (?) detail the "technical" reasons why (I think errors in managing cooperative multi-tasking, which async is - and never spotting the real problem), and I gasped, and so started some side discussions with several of you. So - to everyone - thanks for all the discussion, and thinking, and presenting, and break-away libraries. I hope we keep kicking this ball around a bit more, into new, deeper space! - Yarko On Mon, Aug 8, 2016 at 7:26 AM, David Beazley wrote: > > > On Aug 8, 2016, at 4:45 AM, Cory Benfield wrote: > >> > > > > The first is that, by sheer good luck, I?ve managed to tap into a > zeitgeist and be in the right time at the right place to deliver this > message. Dave Beazley?s work on curio here is helping, because of curio?s > sheer incompatibility with the other event loop approaches, which means > that his work and mine have a nice symbiosis. Nathaniel and I have managed > to give him the building blocks to demonstrate curio?s effectiveness > without him needing to be an expert in HTTP. > > > > Chiming in on the "zeitgeist" comment for a moment, I've wondered for a > long time why Python can't reinvent itself in the area of I/O (and maybe > systems programming generally). Honestly, I feel like a whole lot of > time has been burned up thinking about Python 2/3 compatibility instead of > looking forward with futuristic new projects and ideas. Perhaps > "async/await" serves as a catalyst to rethink some of these things. A lot > of my work with async/await is really focused on exploring the API space > with it--well, at least seeing how much I can twist that part of the > language in diabolical ways. The protocol issue is real though. Sure, I > could probably bang out a passable HTTP/0.9 protocol in an afternoon, but > try to tackle something modern like HTTP/2? No way. I'm totally out of > my element with something like that. Having an I/O-free implementation of > it is cool. It would be pretty neat to have something like that for > various other things too (Redis, MySQL, postgres, ZeroMQ, etc.). > > Cheers, > Dave > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Aug 8 11:47:27 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 8 Aug 2016 08:47:27 -0700 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> Message-ID: OK, never mind Impostor Syndrome... How can we move this forward in the community though actual applications? Should we have the complete I/O-free HTTP parser in the stdlib, or is that a bad idea? Do we want a PEP? Is there something we could add to PEP 8? On Mon, Aug 8, 2016 at 2:45 AM, Cory Benfield wrote: > On 8 Aug 2016, at 07:56, Yarko Tymciurak wrote: > > Question: isn't SansIO / Corey's work just a specific instance of Bob > Martin's "Clean Architecture"? It sounds familiar to me, when thinking of > Brandon Rhode's 2014 PyOhio talk, and his recast of the topic in Warsaw in > 2015. It seems like it... If so, then perhaps async aspects are just a > second aspect. > > > Yeah, so this is worth highlighting more clearly. =) I apologise to the list > in advance for this, but this is going to be something of a digression, so I > have re-titled the thread. > > I have invented and pioneered *nothing* here. Designing protocol > implementations this way is not new in the world of software development. > No-one is going to give me a medal or name a design pattern after me, and I > don?t even think that pushing for this design pattern will help me move up > my employer?s technical career path the way spending time on writing RFCs > would have. > > However, I think we can safely say that the Python community has not > effectively done this over our twenty-plus year lifetime. Brett?s ?sans-io? > page is a beautiful example of this: currently we can find maybe *three* > libraries that tick the box of being I/O-independent. This, in my mind, > points to a cultural problem in the Python community. For whatever reason, > we have tricked ourselves into abandoning what I would call ?good software > design?. > > I have my theories for why this is. A huge factor of this is the relative > ease-of-use of sockets and other I/O tools in Python (relative to C, that > is: languages that also have this relative ease of use have similar > problems, such as Javascript and Ruby). Another factor is the Python > community?s love of beautiful APIs. Libraries with beautiful APIs provide > such an appeal to programmers that there is huge incentive to build such a > library. Building such a library requires including your I/O, and in the > absence of any I/O-free implementations of a protocol you are almost > certainly going to take the path of least resistance, which means > intermingling your I/O and event loop primitives with your protocol code. > > And I?m not criticising programmers that do this. As I pointed out with the > fake title of my talk, I am a programmer that does this! Almost all the best > programmers our community has to offer have done this and continue to do > this. It really is easier. And trust me, as someone who has written one: > there is very little that is sexy about writing a protocol library that does > no I/O. It doesn?t make you famous, and it doesn?t make anyone applaud your > design sense. They?re extremely boring and prosaic, and they rarely solve > the actual problem you have: they are a step on the road to solving the > problem generally, but not specifically. > > So am I talking about the same thing as Brandon? Yes, yes I am. And Brandon > is a better speaker than I am, so I agree that it?s weird that I?m getting > more credit for this than he is. But I suspect that I?ve had some advantages > that Brandon didn?t have. > > The first is that, by sheer good luck, I?ve managed to tap into a zeitgeist > and be in the right time at the right place to deliver this message. Dave > Beazley?s work on curio here is helping, because of curio?s sheer > incompatibility with the other event loop approaches, which means that his > work and mine have a nice symbiosis. Nathaniel and I have managed to give > him the building blocks to demonstrate curio?s effectiveness without him > needing to be an expert in HTTP. > > The second is that, I think, Brandon was targeting a different audience. > Brandon was trying to talk to the general case, but he stopped short of the > case I made. If you go back to watch Brandon?s talk, he talks about hoisting > your I/O to the top of your control flow, but never leaps forward to say > ?and this has a very specific application to any tool or library that talks > a network protocol?. My argument is actually *more specific* than Brandon?s: > I am saying that you can strip out the low-level stuff into an entirely > separate logical codebase, and then not just change the *type* of I/O you > do, but change the complete style in which you do it. Brandon?s argument > isn?t so much about code reuse as it is about code maintenance. > > That said, I think the real reason people are talking about my work and not > Brandon?s is because, while we both said ?You should hoist your I/O out of > your logic?, I got to follow up with ?and I have done it and released the > code and it?s actually being used today in projects you know about?. That > second argument really does help, because whenever you tell people to > totally change the way they?re writing something, they have a tendency to go > ?Oh, yeah, but that doesn?t work for *my* project?. Being able to say ?look, > I did it, it really does work, and here are all the ways that it let me > build a better thing?. > > That said, I?d love to have Brandon weigh in on this too. I do feel a bit > bad that I am re-treading ground that others have covered before, but?hell, > we can?t all be top-tier programmers! If all I say is that I have repackaged > an idea my intellectual betters have previously made such that it?s more > appealing to the masses, I think I can call myself happy with that. > > Cory > > > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From cory at lukasa.co.uk Mon Aug 8 11:56:51 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 8 Aug 2016 16:56:51 +0100 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> Message-ID: > On 8 Aug 2016, at 16:47, Guido van Rossum wrote: > > OK, never mind Impostor Syndrome... How can we move this forward in > the community though actual applications? Should we have the complete > I/O-free HTTP parser in the stdlib, or is that a bad idea? Do we want > a PEP? Is there something we could add to PEP 8? Addressing those in turn: Yes, we could put the I/O free HTTP parser in the stdlib. That?s really Nathaniel?s call, of course, as it?s his parser, but there?s no reason we couldn?t. Of course, all the regular caveats apply: we?ll want to give it a while longer on PyPI to bake, and of course we have to discuss where the logical end of this. How obscure does a protocol have to be for having a parser in the stdlib to no longer be sensible? Having a parser in the stdlib also raises another question: does the http module get rewritten in terms of this parser? The obvious answer is ?yes?, but someone would have to do the work, and I?m not sure who is volunteering (though there?s a risk that it?d be me). As to having a PEP or putting something in PEP 8, I feel pretty lukewarm to those ideas. If the core Python team was able to legislate good coding practices via PEPs I think the world would be a very different place. Instead, it might be a better idea to focus our writing efforts on the SansIO page in the first instance. If we get to a place where we feel like we have a really good handle on how to explain what to do and what not to do, we could reconsider the PEP at that point. Cory From yselivanov at gmail.com Mon Aug 8 12:06:00 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Mon, 8 Aug 2016 12:06:00 -0400 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> Message-ID: <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> > On Aug 8, 2016, at 11:56 AM, Cory Benfield wrote: > > >> On 8 Aug 2016, at 16:47, Guido van Rossum wrote: >> >> OK, never mind Impostor Syndrome... How can we move this forward in >> the community though actual applications? Should we have the complete >> I/O-free HTTP parser in the stdlib, or is that a bad idea? Do we want >> a PEP? Is there something we could add to PEP 8? > > Addressing those in turn: > > Yes, we could put the I/O free HTTP parser in the stdlib. That?s really Nathaniel?s call, of course, as it?s his parser, but there?s no reason we couldn?t. Of course, all the regular caveats apply: we?ll want to give it a while longer on PyPI to bake, and of course we have to discuss where the logical end of this. How obscure does a protocol have to be for having a parser in the stdlib to no longer be sensible? > > Having a parser in the stdlib also raises another question: does the http module get rewritten in terms of this parser? The obvious answer is ?yes?, This ? we have to validate that h11 has the correct design *before* including it to the stdlib. So the answer is ?yes?, indeed. > > As to having a PEP or putting something in PEP 8, I feel pretty lukewarm to those ideas. If the core Python team was able to legislate good coding practices via PEPs I think the world would be a very different place. What does this mean? A lot of people, in any serious project at least, follow PEP 8 and coding with linters is a wide-spread practice. Yury From cory at lukasa.co.uk Mon Aug 8 12:16:25 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 8 Aug 2016 17:16:25 +0100 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> Message-ID: > On 8 Aug 2016, at 17:06, Yury Selivanov wrote: >> >> As to having a PEP or putting something in PEP 8, I feel pretty lukewarm to those ideas. If the core Python team was able to legislate good coding practices via PEPs I think the world would be a very different place. > > What does this mean? A lot of people, in any serious project at least, follow PEP 8 and coding with linters is a wide-spread practice. Sure, but that only works for things that linters can enforce, and I sincerely doubt that this is one of them. How would you code a linter in order to confirm that protocol code and I/O code are separated? More importantly though, people follow the bits of PEP 8 that are easy to enforce and that don?t require substantive architectural changes. If PEP 8 said something like ?wherever possible use dependency injection as a design pattern?, that guideline would be ignored as both entirely unenforceable and totally subjective. Where is the line for ?wherever possible?? How does one enforce the use of dependency injection? Can you programmatically determine dependency injection? What dependencies do not need to be injected? Dependency injection is a great design pattern that produces lots of useful side effects, and I use it often. But if I saw PEP 8 mandating it I?d be extremely perplexed. Realistically, at a certain point it?s the equivalent of writing ?You should write good, maintainable code, obeying all relevant best practices? into PEP 8. *Of course* you should. This goes without saying. But that makes the advice not that helpful. Now, PEP 8 could *recommend* design patterns to follow, and that?s a whole other kettle of fish. But then we?re just asking a different question: how universally praised does a design pattern have to be to become part of PEP 8? Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov at gmail.com Mon Aug 8 12:33:25 2016 From: yselivanov at gmail.com (Yury Selivanov) Date: Mon, 8 Aug 2016 12:33:25 -0400 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> Message-ID: <60D4B62D-3998-4B5B-9A6F-D4A4BC23A572@gmail.com> > On Aug 8, 2016, at 12:16 PM, Cory Benfield wrote: > > >> On 8 Aug 2016, at 17:06, Yury Selivanov wrote: >>> >>> As to having a PEP or putting something in PEP 8, I feel pretty lukewarm to those ideas. If the core Python team was able to legislate good coding practices via PEPs I think the world would be a very different place. >> >> What does this mean? A lot of people, in any serious project at least, follow PEP 8 and coding with linters is a wide-spread practice. > > Sure, but that only works for things that linters can enforce, and I sincerely doubt that this is one of them. Correct. I don?t think it should be PEP 8 either. I think Guido?s idea on including h11 to the stdlib is cool, and that?s the better way to send the message. We can also add a *new* informational PEP instructing people on why (and how) they should write IO free protocol implementations. It would be great if you have time to champion such a PEP (I?d be glad to help if needed.) The other problem with sans-IO approach is that it takes longer to implement properly. You need at least a synchronous and an asynchronous versions to make sure that you got the design parts ?right?. For instance, just few days ago we?ve open sourced a new PostgreSQL driver [1], and right now it?s bound to asyncio. While the foundational layer of the protocol is IO free, it would still take me a lot of time to (a) document it, (b) to build and test anything but asyncio on top of it. And, since our driver is far more feature packed (and faster!) than psycopg2, I?d love to make a synchronous version of it (but don?t have time atm). So what I was thinking about is a library that would provide a set of base Protocol (meta?) classes, one for asyncio, one for sync io, etc. Then the only thing you would need is to mixin your sans-IO protocol to them and write some glue code. Timeouts, cancellations, etc will be already taken care of for you. This is something that is super hard to make for protocols like HTTP, but shouldn?t be a big problem for DB protocols (redis, postgres, etc). Something to think about :) [1] http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/ Thanks, Yury > How would you code a linter in order to confirm that protocol code and I/O code are separated? > > More importantly though, people follow the bits of PEP 8 that are easy to enforce and that don?t require substantive architectural changes. If PEP 8 said something like ?wherever possible use dependency injection as a design pattern?, that guideline would be ignored as both entirely unenforceable and totally subjective. Where is the line for ?wherever possible?? How does one enforce the use of dependency injection? Can you programmatically determine dependency injection? What dependencies do not need to be injected? > > Dependency injection is a great design pattern that produces lots of useful side effects, and I use it often. But if I saw PEP 8 mandating it I?d be extremely perplexed. Realistically, at a certain point it?s the equivalent of writing ?You should write good, maintainable code, obeying all relevant best practices? into PEP 8. *Of course* you should. This goes without saying. But that makes the advice not that helpful. > > Now, PEP 8 could *recommend* design patterns to follow, and that?s a whole other kettle of fish. But then we?re just asking a different question: how universally praised does a design pattern have to be to become part of PEP 8? > > Cory From cory at lukasa.co.uk Mon Aug 8 12:40:43 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 8 Aug 2016 17:40:43 +0100 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <60D4B62D-3998-4B5B-9A6F-D4A4BC23A572@gmail.com> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> <60D4B62D-3998-4B5B-9A6F-D4A4BC23A572@gmail.com> Message-ID: <38A730FB-EDA2-43DC-9D49-D4A1F911F0E7@lukasa.co.uk> > On 8 Aug 2016, at 17:33, Yury Selivanov wrote: > > >> On Aug 8, 2016, at 12:16 PM, Cory Benfield wrote: >> >> >>> On 8 Aug 2016, at 17:06, Yury Selivanov wrote: >>>> >>>> As to having a PEP or putting something in PEP 8, I feel pretty lukewarm to those ideas. If the core Python team was able to legislate good coding practices via PEPs I think the world would be a very different place. >>> >>> What does this mean? A lot of people, in any serious project at least, follow PEP 8 and coding with linters is a wide-spread practice. >> >> Sure, but that only works for things that linters can enforce, and I sincerely doubt that this is one of them. > > Correct. I don?t think it should be PEP 8 either. I think Guido?s idea on including h11 to the stdlib is cool, and that?s the better way to send the message. We can also add a *new* informational PEP instructing people on why (and how) they should write IO free protocol implementations. It would be great if you have time to champion such a PEP (I?d be glad to help if needed.) I am certainly happy to take the lead on a PEP like that if we believe there is value in it. I suspect I?d want to run it past this group quite a few times, because there are many others in this SIG (Brett, Nathaniel, Dave, and Glyph all jump to mind) that would have lots of useful things to say. > The other problem with sans-IO approach is that it takes longer to implement properly. You need at least a synchronous and an asynchronous versions to make sure that you got the design parts ?right?. Nah, that?s not necessary, at least for just the protocol part. You just need a test suite that tests it entirely in memory. Necessarily, if you can test it with tests that don?t involve writing mocks for anything in the socket, select, selectors, or asyncio modules, you?re probably in a pretty good place to be arguing that you?re I/O free. If your tests only use the public interfaces, then you?re totally set. The *outer* layers are where you need to duplicate work, but you don?t need both of those right away. If you have the low-level no-I/O layer in place, you can start with just writing (say) the asyncio implementation and leave the sync implementation for later in the day. I strongly, strongly advocate enforcing this distinction by splitting the no-I/O layer out into an entirely separate Python package that is separately versioned, and that you treat as third-party code from the perspective of your other modules. This has a nice tendency of getting into the headspace of thinking of it as an entirely separate module. In the case of OSS code, it also lets you push out the no-I/O version early: if your implementation really is faster than psycopg2, you might be able to convince someone else to come along and write the sync code instead! Anyway, I?ll stop now, because at a certain point I?ll just start writing that PEP in this email. Cory From gmludo at gmail.com Mon Aug 8 17:11:42 2016 From: gmludo at gmail.com (Ludovic Gasc) Date: Mon, 8 Aug 2016 23:11:42 +0200 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: 2016-08-08 13:24 GMT+02:00 Cory Benfield : > > On 8 Aug 2016, at 11:16, Ludovic Gasc wrote: > > Certainly some protocols/transports should be easier to have this split > than others: Interesting to know if somebody has already tried to have QUIC > and HTTP/2 in the same time with Python. > > > AFAIK they haven?t. This is partly because there?s no good QUIC > implementation to bind from Python at this time. Chromium?s QUIC library > requires a giant pool of custom C++ to bind it appropriately, and Go?s > implementation includes a gigantic runtime and is quite large. > I had the same conclusion. For now, I don't know what's the most complex: Try to do a Python binding or reimplement QUIC in Python ;-) > As and when a good OSS QUIC library starts to surface, I?ll be able to > answer this question more effectively. But I?m not expecting a huge issue. > =) > We'll see when it will happen ;-) Implemented in 2012, pushed on production by Google in 2013, and 3 years later, only one Web browser and one programming language have the support, to my knowledge. Nobody uses that except Google, or everybody already migrated on Go ? ;-) Or simply, it's too much complicated to use/debug/... ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Mon Aug 8 17:15:31 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 8 Aug 2016 22:15:31 +0100 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: > On 8 Aug 2016, at 22:11, Ludovic Gasc wrote: > > We'll see when it will happen ;-) > Implemented in 2012, pushed on production by Google in 2013, and 3 years later, only one Web browser and one programming language have the support, to my knowledge. > Nobody uses that except Google, or everybody already migrated on Go ? ;-) > Or simply, it's too much complicated to use/debug/... ? Well, that?s not entirely true. Akamai have an implementation, as I understand it, though it?s based off the Chromium one. And I expect that others have stacks built in similar ways. My understanding is also that Microsoft are working on an implementation as well. The main reason is just that it?s not ready yet. Google have been taking their time with it, and so it?s been very changeable. In particular, QUIC is moving to use TLS 1.3 as its crypto solution, which is extremely tricky for those of us in Python-land, as OpenSSL currently does not have TLS 1.3 on their roadmap for anytime in the near future. That?ll mean using a different TLS library, which adds an extra wrinkle that is quite inconvenient. So for the medium term I expect this to remain true. FWIW, I?m following the QUIC working group, so I?ll be keeping a very close eye on this over the next few years. Cory From glyph at twistedmatrix.com Mon Aug 8 18:26:02 2016 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 8 Aug 2016 15:26:02 -0700 Subject: [Async-sig] PEP: asynchronous generators In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> Message-ID: <5AAEC9EA-CCE2-4739-B169-7F9A1D6F23D0@twistedmatrix.com> > On Aug 8, 2016, at 2:11 PM, Ludovic Gasc wrote: > > > 2016-08-08 13:24 GMT+02:00 Cory Benfield >: > >> On 8 Aug 2016, at 11:16, Ludovic Gasc > wrote: >> >> Certainly some protocols/transports should be easier to have this split than others: Interesting to know if somebody has already tried to have QUIC and HTTP/2 in the same time with Python. > > AFAIK they haven?t. This is partly because there?s no good QUIC implementation to bind from Python at this time. Chromium?s QUIC library requires a giant pool of custom C++ to bind it appropriately, and Go?s implementation includes a gigantic runtime and is quite large. > > I had the same conclusion. > For now, I don't know what's the most complex: Try to do a Python binding or reimplement QUIC in Python ;-) > > As and when a good OSS QUIC library starts to surface, I?ll be able to answer this question more effectively. But I?m not expecting a huge issue. =) > > We'll see when it will happen ;-) > Implemented in 2012, pushed on production by Google in 2013, and 3 years later, only one Web browser and one programming language have the support, to my knowledge. > Nobody uses that except Google, or everybody already migrated on Go ? ;-) > Or simply, it's too much complicated to use/debug/... ? My understanding is that QUIC was always intended to be an experimental thing mostly (although not entirely) internal to Google. The output of QUIC experimentation has been funneled into standards efforts like TLS 1.3 and HTTP/2. Although I think the protocol as a whole may survive in some form, statements like this one: ?The QUIC crypto protocol is the part of QUIC that provides transport security to a connection. The QUIC crypto protocol is destined to die. It will be replaced by TLS 1.3 in the future, but QUIC needed a crypto protocol before TLS 1.3 was even started.? make it difficult to get excited about implementing the protocol as it stands today. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Aug 9 13:35:04 2016 From: brett at python.org (Brett Cannon) Date: Tue, 09 Aug 2016 17:35:04 +0000 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <38A730FB-EDA2-43DC-9D49-D4A1F911F0E7@lukasa.co.uk> References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> <60D4B62D-3998-4B5B-9A6F-D4A4BC23A572@gmail.com> <38A730FB-EDA2-43DC-9D49-D4A1F911F0E7@lukasa.co.uk> Message-ID: On Mon, 8 Aug 2016 at 11:12 Cory Benfield wrote: > > > On 8 Aug 2016, at 17:33, Yury Selivanov wrote: > > > > > >> On Aug 8, 2016, at 12:16 PM, Cory Benfield wrote: > >> > >> > >>> On 8 Aug 2016, at 17:06, Yury Selivanov wrote: > >>>> > >>>> As to having a PEP or putting something in PEP 8, I feel pretty > lukewarm to those ideas. If the core Python team was able to legislate good > coding practices via PEPs I think the world would be a very different place. > >>> > >>> What does this mean? A lot of people, in any serious project at > least, follow PEP 8 and coding with linters is a wide-spread practice. > >> > >> Sure, but that only works for things that linters can enforce, and I > sincerely doubt that this is one of them. > > > > Correct. I don?t think it should be PEP 8 either. I think Guido?s idea > on including h11 to the stdlib is cool, and that?s the better way to send > the message. We can also add a *new* informational PEP instructing people > on why (and how) they should write IO free protocol implementations. It > would be great if you have time to champion such a PEP (I?d be glad to help > if needed.) > > I am certainly happy to take the lead on a PEP like that if we believe > there is value in it. I suspect I?d want to run it past this group quite a > few times, because there are many others in this SIG (Brett, Nathaniel, > Dave, and Glyph all jump to mind) that would have lots of useful things to > say. > The lighter option is we put the equivalent of a PEP on sans-io.rtfd.io instead of my dinky paragraph or three quickly explaining why this is an important concept. That way there's no python-dev discussion and we can update it w/o issue (I'm happy to add more contributors to the GH repo). > > > The other problem with sans-IO approach is that it takes longer to > implement properly. You need at least a synchronous and an asynchronous > versions to make sure that you got the design parts ?right?. > > Nah, that?s not necessary, at least for just the protocol part. You just > need a test suite that tests it entirely in memory. Right, so in the case of asyncpg it would be pulling out the binary protocol parser and having that stand on its own. Then the asyncio part that uses the binary protocol parser you wrote would be what asyncpg is w/ the protocol parser becomings pgparser or something. > Necessarily, if you can test it with tests that don?t involve writing > mocks for anything in the socket, select, selectors, or asyncio modules, > you?re probably in a pretty good place to be arguing that you?re I/O free. > If your tests only use the public interfaces, then you?re totally set. > Which "public interfaces" are you referring to? For me, any I/O-free library shouldn't be driving the I/O, just a producer/consumer of data. That means even if something follows e.g. the public interface of a socket that it wouldn't qualify as that suggests the library gets to make the call on when the I/O occurs and that you expose a socket for it to use. > > The *outer* layers are where you need to duplicate work, but you don?t > need both of those right away. If you have the low-level no-I/O layer in > place, you can start with just writing (say) the asyncio implementation and > leave the sync implementation for later in the day. > The way I think of it is you make the I/O-free library do as much as possible that allows it to be used in synchronous code, asyncio, and curio. Then it's up to people who are interested in supporting those protocols using those frameworks to wrap it as appropriate (if ever, e.g. maybe no one ever cares about SMTP on curio). -Brett > > I strongly, strongly advocate enforcing this distinction by splitting the > no-I/O layer out into an entirely separate Python package that is > separately versioned, and that you treat as third-party code from the > perspective of your other modules. This has a nice tendency of getting into > the headspace of thinking of it as an entirely separate module. In the case > of OSS code, it also lets you push out the no-I/O version early: if your > implementation really is faster than psycopg2, you might be able to > convince someone else to come along and write the sync code instead! > > Anyway, I?ll stop now, because at a certain point I?ll just start writing > that PEP in this email. > > Cory > > > _______________________________________________ > Async-sig mailing list > Async-sig at python.org > https://mail.python.org/mailman/listinfo/async-sig > Code of Conduct: https://www.python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Tue Aug 9 13:49:34 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 9 Aug 2016 18:49:34 +0100 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> <60D4B62D-3998-4B5B-9A6F-D4A4BC23A572@gmail.com> <38A730FB-EDA2-43DC-9D49-D4A1F911F0E7@lukasa.co.uk> Message-ID: <487C0482-89FC-47C1-8670-CD8894E073EF@lukasa.co.uk> > On 9 Aug 2016, at 18:35, Brett Cannon wrote: > >> Necessarily, if you can test it with tests that don?t involve writing mocks for anything in the socket, select, selectors, or asyncio modules, you?re probably in a pretty good place to be arguing that you?re I/O free. If your tests only use the public interfaces, then you?re totally set. > > Which "public interfaces" are you referring to? For me, any I/O-free library shouldn't be driving the I/O, just a producer/consumer of data. That means even if something follows e.g. the public interface of a socket that it wouldn't qualify as that suggests the library gets to make the call on when the I/O occurs and that you expose a socket for it to use. Sorry, I just meant the public API of the no-I/O library. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Aug 11 08:00:42 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 11 Aug 2016 13:00:42 +0100 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: References: <63CC2751-BFC2-49F2-808F-A72D059D3F42@gmail.com> <149BEF4D-41C6-4523-95D1-31C5E1B0183D@gmail.com> <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <78178C41-D101-464F-917A-E1C1E5A4C98F@gmail.com> <60D4B62D-3998-4B5B-9A6F-D4A4BC23A572@gmail.com> <38A730FB-EDA2-43DC-9D49-D4A1F911F0E7@lukasa.co.uk> Message-ID: > On 9 Aug 2016, at 18:35, Brett Cannon wrote: > > The lighter option is we put the equivalent of a PEP on sans-io.rtfd.io instead of my dinky paragraph or three quickly explaining why this is an important concept. That way there's no python-dev discussion and we can update it w/o issue (I'm happy to add more contributors to the GH repo). > Alright, enough talking, let?s start with the doing. Here is a GitHub pull request that begins to shape out a document like an informational proto-PEP on sans-io.rtfd.org : https://github.com/brettcannon/sans-io/pull/4 . This document is absolutely early work, and I?d love for this SIG to provide as much feedback as possible on it, as well as extra text, or elaborations, or requests for clarification, or anything else. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Thu Aug 11 12:56:39 2016 From: antoine at python.org (Antoine Pitrou) Date: Thu, 11 Aug 2016 18:56:39 +0200 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> References: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> Message-ID: <57ACAE47.5020100@python.org> Hi, > However, I think we can safely say that the Python community has not > effectively done this over our twenty-plus year lifetime. I'd like to offer a couple remarks here: 1) implementing a protocol usually goes beyond parsing (which, it's true, can easily be done "sans IO"). 2) many non-trivial protocols are stateful, at least at the level of a single connection; the statefulness may require doing I/O spontaneously (example: sending a keepalive packet). You can partly solve this by having a lower layer implementing the stateless parts ("sans IO") and an upper layer implementing the rest above it, but depending on the protocol it may be impossible to offer an *entire* implementation that doesn't depend on at least some notion of I/O. 3) the Protocol abstraction in asyncio (massively inspired from Twisted, of course) is a pragmatic way to minimize the I/O coupling of protocol implementations (and one of the reasons why I pushed for it during the PEP discussion): it still has some I/O-related elements to it (a couple callbacks on Protocol, and a couple methods on Transport), but in a way that makes ignoring them much easier than when using "streams", sockets or similar concepts. Regards Antoine. From brett at python.org Thu Aug 11 14:01:28 2016 From: brett at python.org (Brett Cannon) Date: Thu, 11 Aug 2016 18:01:28 +0000 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <57ACAE47.5020100@python.org> References: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <57ACAE47.5020100@python.org> Message-ID: On Thu, 11 Aug 2016 at 10:23 Antoine Pitrou wrote: > > Hi, > > > However, I think we can safely say that the Python community has not > > effectively done this over our twenty-plus year lifetime. > > I'd like to offer a couple remarks here: > > 1) implementing a protocol usually goes beyond parsing (which, it's > true, can easily be done "sans IO"). > Yes, but parsing the data is at least one of the steps that people have historically not factored out into a stand-alone library. > > 2) many non-trivial protocols are stateful, at least at the level of a > single connection; the statefulness may require doing I/O spontaneously > (example: sending a keepalive packet). You can partly solve this by > having a lower layer implementing the stateless parts ("sans IO") and an > upper layer implementing the rest above it, but depending on the > protocol it may be impossible to offer an *entire* implementation that > doesn't depend on at least some notion of I/O. > Also true. While this can either be handled by a state machine emitting keepalive events or simply telling people they may need to emit something, this doesn't detract from the fact that at least parsing the data off the wire can be done as a standalone project (making a state machine that works for the protocol w/o any I/O will vary from protocol to protocol). > > 3) the Protocol abstraction in asyncio (massively inspired from Twisted, > of course) is a pragmatic way to minimize the I/O coupling of protocol > implementations (and one of the reasons why I pushed for it during the > PEP discussion): it still has some I/O-related elements to it (a couple > callbacks on Protocol, and a couple methods on Transport), but in a way > that makes ignoring them much easier than when using "streams", sockets > or similar concepts. > Yep. Once again, no one is saying that a I/O-free approach to protocols will work in all situations, but it should be considered and in cases where it does work it's good to have and can be used with asyncio's ABCs. IOW you're totally right that I/O-free protocol libraries will not always work, but sometimes they do and people should thus consider structuring their libraries that way when it makes sense. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Fri Aug 12 04:23:35 2016 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 12 Aug 2016 09:23:35 +0100 Subject: [Async-sig] SansIO (Was: PEP: asynchronous generators) In-Reply-To: <57ACAE47.5020100@python.org> References: <4ADA55F6-692F-44F6-9C78-8BE1432AECA0@lukasa.co.uk> <57ACAE47.5020100@python.org> Message-ID: <668EF826-6171-4D3A-8D3A-50A442075A74@lukasa.co.uk> > On 11 Aug 2016, at 17:56, Antoine Pitrou wrote: > > 1) implementing a protocol usually goes beyond parsing (which, it's > true, can easily be done "sans IO?). Yes, no question. And in fact, the hyper-h2 docs are very clear on this point: http://python-hyper.org/projects/h2/en/stable/basic-usage.html You cannot just drop in hyper-h2 and expect anything sensible to happen. It needs to be hooked up to I/O, and the user needs to make some decisions themselves. However, I?m totally happy to stand by my original point, which was: regardless of whether or not writing a protocol parser and state machine is ?easy? to do without I/O, the Python community has not done that for the last twenty years. The fact that the current list of sans-IO implementations is three entries long, two of which are less than a year old, is a good indication of that point. I don?t believe anyone is saying that sans-IO protocol implementations will remove *all* work from writing full-fledged implementations. Such a dream is impossible. But ideally they will remove a large chunk of the work. > 2) many non-trivial protocols are stateful, at least at the level of a > single connection; the statefulness may require doing I/O spontaneously > (example: sending a keepalive packet). You can partly solve this by > having a lower layer implementing the stateless parts ("sans IO") and an > upper layer implementing the rest above it, but depending on the > protocol it may be impossible to offer an *entire* implementation that > doesn't depend on at least some notion of I/O. So the spontaneous I/O question is an interesting one, not because it involves doing I/O so much as because your example fundamentally involves access to a *clock*. I haven?t had to deal with this yet, but I?ve been thinking about it a bit. My current conclusion is that a clock is basically an I/O tool: it?s a thing that needs to be controlled from the outside implementation. This is largely because when we want to use clocks what we really want to do is use *timers*, and timers are a flow control tool. This means that they fit into the category of thing I alluded to briefly in my talk: they?re a thing that the sans-IO implementation can provide help and guidance with, but not ultimately do itself. Another example of this has been flow control management in HTTP/2: while the sans-IO implementation can do a lot of the work, fundamentally it still needs you to tell it ?I just dealt with 100kB of data, please free up that much space in the window?. There is no getting around this for almost any protocol, but that?s ok. Once again, the goal is not to subsume *everything* about a protocol: as you point out, that?s simply not possible. Instead, the goal is to subsume as much as possible. > 3) the Protocol abstraction in asyncio (massively inspired from Twisted, > of course) is a pragmatic way to minimize the I/O coupling of protocol > implementations (and one of the reasons why I pushed for it during the > PEP discussion): it still has some I/O-related elements to it (a couple > callbacks on Protocol, and a couple methods on Transport), but in a way > that makes ignoring them much easier than when using "streams", sockets > or similar concepts. I agree: this is why I used Twisted Protocols in my discussion with Nick in python-ideas. However, they do only *minimize* it. Most asyncio/Twisted Protocols quite happily issue writes to their transports willy-nilly, and also a great many of them create Futures (which makes sense: the abstraction into the coroutine world has to happen somewhere!). Once you create a Future, you are no longer ?sans-IO?: a Future is an event loop construct. (On a side note, this is why Twisted has a slight hypothetical edge in the ?sans-IO? race: an asyncio.Future is an event-loop construct, but a twisted.internet.defer.Deferred is not. Deferreds work perfectly without an event loop, but a Future always requires one.) The biggest problem, though, is that an asyncio Protocol is written like this: class MyProtocol(asyncio.Protocol): pass This provides substantial programmer baggage. Because asyncio has a blessed I/O model, it is very, very difficult for most programmers to think about writing a Protocol that isn?t going to be used that way. Even though, as I demonstrated with Twisted Protocols in python-ideas, they absolutely do not require an event loop if written carefully. This is part of why divorcing your protocol library from asyncio *entirely* (don?t even import it!) is helpful: it forces a clean, clear line in the sand that says ?I do not care how you do I/O?. Twisted has been fighting this battle for years, and asyncio isn?t going to have a better time of it. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: