[Python-ideas] Making concurrent.futures.Futures awaitable

[Nick Coghlan]

On 8 Aug 2015 22:48, "Alex Grönholm" <alex.gronholm at nextday.fi> wrote:
>
> That name would and argument placement would be better, but are you
suggesting that the ability to pass along extra arguments should be
removed? The original method was bad enough in that it only supported
positional and not keyword arguments, forcing users to pass partial()
objects as callables.

That's a deliberate design decision in many of asyncio's APIs to improve
the introspection capabilities and to clearly separate concerns between
"interacting with the event loop" and "the operation being dispatched for
execution".

>> With the suggested change to the method name and signature, the same
>> example would instead look like:
>>
>>      async def handler(self):
>>          loop = asyncio.get_event_loop()
>>          result = await
>> loop.call_in_background(some_blocking_api.some_blocking_call)
>>          await self.write(result)
>
> Am I the only one who's bothered by the fact that you have to get a
reference to the event loop first?
> Wouldn't this be better:
>
> async def handler(self):
>
>     result = await
asyncio.call_in_background(some_blocking_api.some_blocking_call)
>
>     await self.write(result)

That was my original suggestion a few weeks ago, but after playing with it
for a while, I came to agree with Guido that hiding the event loop in this
case likely wasn't helpful to the conceptual learning process. Outside
higher level frameworks that place more constraints on your code, you
really can't get very far with asyncio without becoming comfortable with
interacting with the event loop directly.

I gave a demo using the current spelling as a lightning talk at PyCon
Australia last weekend:
https://www.youtube.com/watch?v=_pfJZfdwkgI

The only part of that demo I really wasn't happy with was the
"run_in_executor" call - the rest all felt good for the level asyncio
operates at, while still allowing higher level third party APIs that hide
more of the underlying machinery (like the event loop itself, as well as
the use of partial function application).
>
>
> The call_in_background() function would return an awaitable object that
is recognized by the asyncio Task class, which would then submit the
function to the default executor of the event loop.
>
>> That should make sense to anyone reading the handler, even if they
>> know nothing about concurrent.futures - the precise mechanics of how
>> the event loop goes about handing off the call to a background thread
>> or process is something they can explore later, they don't need to
>> know about it in order to locally reason about this specific handler.
>>
>> It also means that event loops would be free to implement their
>> *default* background call functionality using something other than
>> concurrent.futures, and only switch to the latter if an executor was
>> specified explicitly.
>
> Do you mean background calls that don't return objects compatible with
concurrent.futures.Futures?

A background call already returns an asyncio awaitable, not a
concurrent.futures.Future object.

> Can you think of a use case for this?

Yes, third party event loops like Twisted may have their own background
call mechanism that they'd prefer to use by default, rather than the
concurrent.futures model.

>> There are still some open questions about whether it makes sense to
>> allow callables to indicate whether or not they expect to be IO bound
>> or CPU bound,
>
> What do you mean by this?

There was a thread on the idea recently, but I don't have a link handy.
Indicating CPU vs IO bound directly wouldn't work (that's context
dependent), but allowing callables to explicitly indicate "recommended",
"supported", "incompatible" for process pools could be interesting.

>>   and hence allow event loop implementations to opt to
>> dispatch the latter to a process pool by default
>
> Bad idea! The semantics are too different and process pools have too many
limitations.

Yes, that's why I find it an intriguing notion to allow callables to
explicitly indicate whether or not they're compatible with them.

Cheers,
Nick
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150809/e205da32/attachment-0001.html>