[Async-sig] Warts both in asyncio way and greenlet-based approach

[Glyph Lefkowitz]

On Aug 4, 2016, at 4:47 PM, Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
> 
> @cory I didn't want to say "re-implementing requests in asyncio way is impossible".
> 
> I just want to say: it's hard.

I can't comment on the total difficulty as I'm not a requests maintainer, but I think you're overestimating it.

> Requires massive code rewriting.

In fact, requests had an async mode in the past, and if you look at the internal factoring, there are a totally manageable number of places where it actually depends on blocking.  It would not be a "massive" rewrite, just a refactoring around those integration points between layers.

> Moreover, it's pain to support both Python 2 and 3 with asyncio inthe same code base.

Not really.  You can't use the asyncio task scheduler, but we've been upgrading to be able to support asyncio from Twisted and it's a lot easier than many other Python 3 porting tasks :).

> Really only after dropping python 2 support you are able to adopt requests for asyncio.

Also not correct; we (Twisted) have supported many python3 features (for example; 'return' out of @inlineCallbacks-decorated generators) for several years despite being on python 2; nothing about the _interfaces_ that you have to support in order for asyncio users to use it are in any way version-specific.

> Hopefully after 2020.

We should be so lucky :).

> Requests library (as well as sqlalchemy) are very crucial scaffolds for python community, I suspect you will be forced to support Python 2 up to the official death of branch at least.

I don't think that's in question.

-g

> On Wed, Aug 3, 2016 at 5:27 PM Cory Benfield <cory at lukasa.co.uk <mailto:cory at lukasa.co.uk>> wrote:
> 
>> On 2 Aug 2016, at 19:39, Andrew Svetlov <andrew.svetlov at gmail.com <mailto:andrew.svetlov at gmail.com>> wrote:
>> 
>> Long story short: in async world every IO library should use asyncio.
>> Unfortunately for the most  popular libraries (requests, django, flask, sqlalchemy etc.) it's impossible to rewrite the code keeping backward compatibility. Much easier to create a new library than rewrite existing one.
> 
> Removing my hyper hat for a moment and replacing it with my Requests hat: we disagree. Some of the discussion has happened on GitHub at [1], but I can summarise it again here.
> 
> One of the nice things with an event loop is that you can retrofit synchronous code on top of it. That is, if you have an event loop, you can always turn your evented code (code that returned a Deferred (or Deferred-alike) or a coroutine) into synchronous code by simply running the event loop in the synchronous call. That is, given a method signature like this:
> 
> async def get(url):
>     return await do_the_work(url)
> 
> It can be turned into this:
> 
> def sync_get(url):
>     return some_event_loop.run(get(url))
> 
> The reason Requests has not previously done this is that it required us to do one of two things. In the first instance, we could bundle or depend on an event loop (e.g. Twisted’s reactors). That’s annoying: it’s a lot of code that we don’t really care about in order to achieve a task that is fundamentally an implementation detail. The other option is to write backends for all possible event loops *and* a kind of event-loop-alike synchronous approach that can be run as a series of coroutines, but that is actually entirely synchronous under the covers (essentially, ship a coroutine runner that never waits for anything). Both of these were a lot of work for fairly minimal gain.
> 
> However, with asyncio’s event loop becoming shipped with the language, and with the other event loop implementations shimming in compatibility layers, libraries like Requests have a perfect escape hatch. Now we only need *two* backends: one for the asyncio event loop, and one synchronous one. And if we’re prepared to depend on Python 3.4+, we only need *one* backend: asyncio, because we can rely on it being present.
> 
> This drastic reduction in the space of event loops we need to support suddenly makes it much more viable to consider adjusting the way we do I/O. There’s still a *lot* of work there, and no-one has actually started sitting down to do the hard stuff, but it’s certainly not impossible any longer.
> 
> Cory
> 
> [1]: https://github.com/kennethreitz/requests/issues/1390#issuecomment-224772923 <https://github.com/kennethreitz/requests/issues/1390#issuecomment-224772923>
> -- 
> Thanks,
> Andrew Svetlov
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org <mailto:Async-sig at python.org>
> https://mail.python.org/mailman/listinfo/async-sig <https://mail.python.org/mailman/listinfo/async-sig>
> Code of Conduct: https://www.python.org/psf/codeofconduct/ <https://www.python.org/psf/codeofconduct/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160804/8b363325/attachment-0001.html>