[Async-sig] "Coroutines" sometimes run without being scheduled on an event loop

[Andrew Svetlov]

What real problem do you want to solve?
Correct code should always use `await loop.sock_connect(sock, addr)`, it
this case the behavior difference never hurts you.

On Thu, May 3, 2018 at 7:04 PM twisteroid ambassador <
twisteroid.ambassador at gmail.com> wrote:

> Hi,
>
> tl;dr: coroutine functions and regular functions returning Futures
> behave differently: the latter may start running immediately without
> being scheduled on a loop, or even with no loop running. This might be
> bad since the two are sometimes advertised to be interchangeable.
>
>
> I find that sometimes I want to construct a coroutine object, store it
> for some time, and run it later. Most times it works like one would
> expect: I call a coroutine function which gives me a coroutine object,
> I hold on to the coroutine object, I later await it or use
> loop.create_task(), asyncio.gather(), etc. on it, and only then it
> starts to run.
>
> However, I have found some cases where the "coroutine" starts running
> immediately. The first example is loop.run_in_executor(). I guess this
> is somewhat unsurprising since the passed function don't actually run
> in the event loop. Demonstrated below with strace and the interactive
> console:
>
> $ strace -e connect -f python3
> Python 3.6.5 (default, Apr  4 2018, 15:01:18)
> [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import asyncio
> >>> import socket
> >>> s = socket.socket()
> >>> loop = asyncio.get_event_loop()
> >>> coro = loop.sock_connect(s, ('127.0.0.1', 80))
> >>> loop.run_until_complete(asyncio.sleep(1))
> >>> task = loop.create_task(coro)
> >>> loop.run_until_complete(asyncio.sleep(1))
> connect(3, {sa_family=AF_INET, sin_port=htons(80),
> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection
> refused)
> >>> s.close()
> >>> s = socket.socket()
> >>> coro2 = loop.run_in_executor(None, s.connect, ('127.0.0.1', 80))
> strace: Process 13739 attached
> >>> [pid 13739] connect(3, {sa_family=AF_INET, sin_port=htons(80),
> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)
>
> >>> coro2
> <Future pending cb=[_chain_future.<locals>._call_check_cancel() at
> /usr/lib64/python3.6/asyncio/futures.py:403]>
> >>> loop.run_until_complete(asyncio.sleep(1))
> >>> coro2
> <Future finished exception=ConnectionRefusedError(111, 'Connection
> refused')>
> >>>
>
> Note that with loop.sock_connect(), the connect syscall is only run
> after loop.create_task() is called on the coroutine AND the loop is
> running. On the other hand, as soon as loop.run_in_executor() is
> called on socket.connect, the connect syscall gets called, without the
> event loop running at all.
>
> Another such case is with Python 3.4.2, where even loop.sock_connect()
> will run immediately:
>
> $ strace -e connect -f python3
> Python 3.4.2 (default, Oct  8 2014, 10:45:20)
> [GCC 4.9.1] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import socket
> >>> import asyncio
> >>> loop = asyncio.get_event_loop()
> >>> s = socket.socket()
> >>> c = loop.sock_connect(s, ('127.0.0.1', 82))
> connect(7, {sa_family=AF_INET, sin_port=htons(82),
> sin_addr=inet_addr("127.0.0.1")}, 16) = -1ECONNREFUSED (Connection
> refused)
> >>> c
> <Future finished exception=ConnectionRefusedError(111, 'Connection
> refused')>
> >>>
>
> In both these cases, the misbehaving "coroutine" aren't actually
> defined as coroutine functions, but regular functions returning a
> Future, which is probably why they don't act like coroutines. However,
> coroutine functions and regular functions returning Futures are often
> used interchangeably: Python docs Section 18.5.3.1 even says:
>
> > Note: In this documentation, some methods are documented as coroutines,
> even if they are plain Python functions returning a Future. This is
> intentional to have a freedom of tweaking the implementation of these
> functions in the future.
>
> In particular, both run_in_executor() and sock_connect() are
> documented as coroutines.
>
> If an asyncio API may change from a function returning Future to a
> coroutine function and vice versa any time, then one cannot rely on
> the behavior of creating the "coroutine object" not running the
> coroutine immediately. This seems like an important Gotcha waiting to
> bite someone.
>
> Back to the scenario in the beginning. If I want to write a function
> that takes coroutine objects and schedule them to run later, and some
> coroutine objects turn out to be misbehaving like above, then they
> will run too early. To avoid this, I could either 1. pass the
> coroutine functions and their arguments separately "callback style",
> 2. use functools.partial or lambdas, or 3. always pass in real
> coroutine objects returned from coroutine functions defined with
> "async def". Does this sound right?
>
> Thanks,
>
> twistero
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180503/d7ff5c90/attachment.html>