[Async-sig] "Coroutines" sometimes run without being scheduled on an event loop

Thu May 3 17:25:33 EDT 2018

I doubt if we should specify such things very explicitly. Call it
"implementation detail" :)

FYI in Python 3.7 all `sock_*()` methods are native coroutines now.
`run_in_executor()` is a regular function that returns a future object. I
don't remember is it the only exception or asyncio has other functions with
such return type.

On Thu, May 3, 2018 at 11:56 PM Chris Jerdonek <chris.jerdonek at gmail.com>
wrote:

> It would probably be hard for people to find at this point all the
> places they might be relying on this behavior (if anywhere), but isn't
> this a basic documented property of coroutines?
>
> From the introduction section on coroutines [1]:
>
> > Calling a coroutine does not start its code running – the coroutine
> object returned by the call doesn’t do anything until you schedule its
> execution. There are two basic ways to start it running: call await
> coroutine or yield from coroutine from another coroutine (assuming the
> other coroutine is already running!), or schedule its execution using the
> ensure_future() function or the AbstractEventLoop.create_task() method.
>
> > Coroutines (and tasks) can only run when the event loop is running.
>
> [1]: https://docs.python.org/3/library/asyncio-task.html#coroutines
>
> --Chris
>
> On Thu, May 3, 2018 at 1:24 PM, Guido van Rossum <gvanrossum at gmail.com>
> wrote:
> > Depending on the coroutine*not* running sounds like asking for trouble.
> >
> > On Thu, May 3, 2018, 09:38 Andrew Svetlov <andrew.svetlov at gmail.com>
> wrote:
> >>
> >> What real problem do you want to solve?
> >> Correct code should always use `await loop.sock_connect(sock, addr)`, it
> >> this case the behavior difference never hurts you.
> >>
> >> On Thu, May 3, 2018 at 7:04 PM twisteroid ambassador
> >> <twisteroid.ambassador at gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> tl;dr: coroutine functions and regular functions returning Futures
> >>> behave differently: the latter may start running immediately without
> >>> being scheduled on a loop, or even with no loop running. This might be
> >>> bad since the two are sometimes advertised to be interchangeable.
> >>>
> >>>
> >>> I find that sometimes I want to construct a coroutine object, store it
> >>> for some time, and run it later. Most times it works like one would
> >>> expect: I call a coroutine function which gives me a coroutine object,
> >>> I hold on to the coroutine object, I later await it or use
> >>> loop.create_task(), asyncio.gather(), etc. on it, and only then it
> >>> starts to run.
> >>>
> >>> However, I have found some cases where the "coroutine" starts running
> >>> immediately. The first example is loop.run_in_executor(). I guess this
> >>> is somewhat unsurprising since the passed function don't actually run
> >>> in the event loop. Demonstrated below with strace and the interactive
> >>> console:
> >>>
> >>> $ strace -e connect -f python3
> >>> Python 3.6.5 (default, Apr  4 2018, 15:01:18)
> >>> [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
> >>> Type "help", "copyright", "credits" or "license" for more information.
> >>> >>> import asyncio
> >>> >>> import socket
> >>> >>> s = socket.socket()
> >>> >>> loop = asyncio.get_event_loop()
> >>> >>> coro = loop.sock_connect(s, ('127.0.0.1', 80))
> >>> >>> loop.run_until_complete(asyncio.sleep(1))
> >>> >>> task = loop.create_task(coro)
> >>> >>> loop.run_until_complete(asyncio.sleep(1))
> >>> connect(3, {sa_family=AF_INET, sin_port=htons(80),
> >>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection
> >>> refused)
> >>> >>> s.close()
> >>> >>> s = socket.socket()
> >>> >>> coro2 = loop.run_in_executor(None, s.connect, ('127.0.0.1', 80))
> >>> strace: Process 13739 attached
> >>> >>> [pid 13739] connect(3, {sa_family=AF_INET, sin_port=htons(80),
> >>> >>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED
> (Connection refused)
> >>>
> >>> >>> coro2
> >>> <Future pending cb=[_chain_future.<locals>._call_check_cancel() at
> >>> /usr/lib64/python3.6/asyncio/futures.py:403]>
> >>> >>> loop.run_until_complete(asyncio.sleep(1))
> >>> >>> coro2
> >>> <Future finished exception=ConnectionRefusedError(111, 'Connection
> >>> refused')>
> >>> >>>
> >>>
> >>> Note that with loop.sock_connect(), the connect syscall is only run
> >>> after loop.create_task() is called on the coroutine AND the loop is
> >>> running. On the other hand, as soon as loop.run_in_executor() is
> >>> called on socket.connect, the connect syscall gets called, without the
> >>> event loop running at all.
> >>>
> >>> Another such case is with Python 3.4.2, where even loop.sock_connect()
> >>> will run immediately:
> >>>
> >>> $ strace -e connect -f python3
> >>> Python 3.4.2 (default, Oct  8 2014, 10:45:20)
> >>> [GCC 4.9.1] on linux
> >>> Type "help", "copyright", "credits" or "license" for more information.
> >>> >>> import socket
> >>> >>> import asyncio
> >>> >>> loop = asyncio.get_event_loop()
> >>> >>> s = socket.socket()
> >>> >>> c = loop.sock_connect(s, ('127.0.0.1', 82))
> >>> connect(7, {sa_family=AF_INET, sin_port=htons(82),
> >>> sin_addr=inet_addr("127.0.0.1")}, 16) = -1ECONNREFUSED (Connection
> >>> refused)
> >>> >>> c
> >>> <Future finished exception=ConnectionRefusedError(111, 'Connection
> >>> refused')>
> >>> >>>
> >>>
> >>> In both these cases, the misbehaving "coroutine" aren't actually
> >>> defined as coroutine functions, but regular functions returning a
> >>> Future, which is probably why they don't act like coroutines. However,
> >>> coroutine functions and regular functions returning Futures are often
> >>> used interchangeably: Python docs Section 18.5.3.1 even says:
> >>>
> >>> > Note: In this documentation, some methods are documented as
> coroutines,
> >>> > even if they are plain Python functions returning a Future. This is
> >>> > intentional to have a freedom of tweaking the implementation of these
> >>> > functions in the future.
> >>>
> >>> In particular, both run_in_executor() and sock_connect() are
> >>> documented as coroutines.
> >>>
> >>> If an asyncio API may change from a function returning Future to a
> >>> coroutine function and vice versa any time, then one cannot rely on
> >>> the behavior of creating the "coroutine object" not running the
> >>> coroutine immediately. This seems like an important Gotcha waiting to
> >>> bite someone.
> >>>
> >>> Back to the scenario in the beginning. If I want to write a function
> >>> that takes coroutine objects and schedule them to run later, and some
> >>> coroutine objects turn out to be misbehaving like above, then they
> >>> will run too early. To avoid this, I could either 1. pass the
> >>> coroutine functions and their arguments separately "callback style",
> >>> 2. use functools.partial or lambdas, or 3. always pass in real
> >>> coroutine objects returned from coroutine functions defined with
> >>> "async def". Does this sound right?
> >>>
> >>> Thanks,
> >>>
> >>> twistero
> >>> _______________________________________________
> >>> Async-sig mailing list
> >>> Async-sig at python.org
> >>> https://mail.python.org/mailman/listinfo/async-sig
> >>> Code of Conduct: https://www.python.org/psf/codeofconduct/
> >>
> >> --
> >> Thanks,
> >> Andrew Svetlov
> >> _______________________________________________
> >> Async-sig mailing list
> >> Async-sig at python.org
> >> https://mail.python.org/mailman/listinfo/async-sig
> >> Code of Conduct: https://www.python.org/psf/codeofconduct/
> >
> >
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
> >
>
-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180503/be481393/attachment-0001.html>