concurrent futures, async futures and await

[Nagy László Zsolt]

> My guess: because asyncio wouldn't know what to do with a
> concurrent.futures.Future.
I see that as a problem.
>
> The tornado docs say that "You can also use
> tornado.gen.convert_yielded to convert anything that would work with
> yield into a form that will work with await":
> http://www.tornadoweb.org/en/stable/guide/coroutines.html#python-3-5-async-and-await
It is true if you yield it from a normal tornado.gen.coroutine, and then
let tornado's own ioloop handle it. But if you try to refactor to a
native coroutine and await for it (instead of yield), then it will stop
working. It is because the await statement is not implemented by
tornado. It is implemented in core Python, and it does not support a
concurrent futures. But hey, concurrent futures are part of the standard
library, so they *should* work together.

Here is an example that makes the problem clear.

This one works:

executor = ThreadPoolExecutor(4)

@tornado.gen.coroutine
def produce_chunks(filename, chunk_size=8192):
        with open(filename,"rb") as fin:
            chunk = yield executor.submit(f.read, chunk_size)
            process_chunk(chunk)
  

It is because "yield executor.submit" will yield the concurrent future
to the ioloop's handler, and tornado's ioloop is clever enough to detect
it, and wrap it in a thread-safe way.

But this won't work:

async def produce_chunks(filename, chunk_size=8192):
    with open(filename,"rb") as fin:
        chunk = await executor.submit(f.read, chunk_size)
        process_chunk(chunk)
  

Simply because the concurrent future returned by executor.submit does
not implement __await__ and so it cannot be awaited for.

Right now asyncio does not know how to handle this. But I think it
should, because it is part of the standard library, and there are very
important use cases (like the one above) that cannot be implemented
without concurrent futures. In the above example, file.read will almost
certainly block the execution, and there is no platform independent way
of doing an async file read operation other than doing it in a different
thread. AFAIK async file reads are not supported in the Linux kernel,
and the only way to do this is to use a thread.

Of course, asyncio should not care if the executor is doing the task in
a different thread or a different process. All I'm saying is that
concurrent.futures.Future should implement the __await__ method, and
asyncio should be able to use it.