Question about asyncio and blocking operations

[Ian Kelly]

On Jan 25, 2016 2:04 AM, "Frank Millman" <frank at chagford.com> wrote:
>
> "Ian Kelly"  wrote in message
news:CALwzidnGOgpx+CpMVBA8vpEFuq4-BwMVS0gZ3ShB0oWZi0Bw+Q at mail.gmail.com...
>>
>> This seems to be a common misapprehension about asyncio programming.
>> While coroutines are the focus of the library, they're based on
>> futures, and so by working at a slightly lower level you can also
>> handle them as such. So  while this would be the typical way to use
>> run_in_executor:
>>
>> async def my_coroutine(stuff):
>>     value = await get_event_loop().run_in_executor(None,
>> blocking_function, stuff)
>>     result = await do_something_else_with(value)
>>     return result
>>
>> This is also a perfectly valid way to use it:
>>
>> def normal_function(stuff):
>>     loop = get_event_loop()
>>     coro = loop.run_in_executor(None, blocking_function, stuff)
>>     task = loop.create_task(coro)
>>     task.add_done_callback(do_something_else)
>>     return task
>
>
> I am struggling to get my head around this.
>
> 1. In the second function, AFAICT coro is already a future. Why is it
necessary to turn it into a task? In fact when I tried that in my testing,
I got an assertion error -
>
> File: "C:\Python35\lib\asyncio\base_events.py", line 211, in create_task
>    task = tasks.Task(coro, loop=self)
> File: "C:\Python35\lib\asyncio\tasks.py", line 70, in __init__
>    assert coroutines.iscoroutine(coro), repr(coro)
> AssertionError: <Future pending ... >

I didn't test this; it was based on the documentation, which says that
run_in_executor is a coroutine. Looking at the source, it's actually a
function that returns a future, so this may be a documentation bug.

There's no need to get a task specifically. We just need a future so that
callbacks can be added, so if the result of run_in_executor is already a
future then the create_task call is unnecessary. To be safe, you could
replace that call with asyncio.ensure_future, which accepts any awaitable
and returns a future.

> 2. In the first function, calling 'run_in_executor' unblocks the main
loop so that it can continue with other tasks, but the function itself is
suspended until the blocking function returns. In the second function, I
cannot see how the function gets suspended. It looks as if the blocking
function will run in the background, and the main function will continue.

Correct. It's not a coroutine, so it has no facility for being suspended
and resumed; it can only block or return. That's why the callback is
necessary to schedule additional code to run after blocking_function
finishes. normal_function itself can continue to make other non-blocking
calls such as scheduling additional tasks, but it shouldn't do anything
that depends on the result of blocking_function since it can't be assumed
to be available yet.

> I would like to experiment with this further, but I would need to see the
broader context - IOW see the 'caller' of normal_function(), and see what
it does with the return value.

The caller of normal_function can do anything it wants with the return
value, including adding additional callbacks or just discarding it. The
caller could be a coroutine or another normal non-blocking function. If
it's a coroutine, then it can await the future, but it doesn't need to
unless it wants to do something with the result. Depending on what the
future represents, it might also be considered internal to normal_function,
in which case it shouldn't be returned at all.