From yselivanov at gmail.com  Fri Jun  3 16:11:51 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Fri, 3 Jun 2016 16:11:51 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
Message-ID: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>

During the asyncio open space a few days ago we had a number of important questions raised, including Twisted/asyncio interop.  I?d like to start the discussion of a few related things:

1. key difference between a Deferred and a Future;
2. make Deferred objects compatible with asyncio code;
3. overall strategy to make Twisted code compatible with asyncio code.

In detail:

1. Unlike Futures, Deferred objects know nothing about the event loop.  Deferred calls its callbacks directly, while Future uses ?loop.call_soon? to ensure fair scheduling.

2. I see two issues with Deferred/Future compatibility:

2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ?asyncio.get_running_loop? function.  If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.

2.2. asyncio uses ?isinstance(o, Future)? calls in many key places, such as ?asyncio.ensure_future? and in the Task class.  The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code.  We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects.  The former option will not be considered because it makes isinstance calls much slower.

As for the second option ? a protocol for Future-like objects ? I propose to add a new dunder attribute ?__asyncio_future__ = True? to asyncio.Future class, and replace all ?isinstance(o, Future)? calls with ?hasattr(o, ?__asyncio_future__?)? checks.

(2.1) and (2.2) should make it possible for Twisted to make Deferred objects fully compatible with asyncio code.  Glyph and Amber, please confirm this.

3. I?ve been on Amber?s talk about Twisted and asyncio, and wanted to share my ideas on how Twisted should approach compatibility with asyncio and async/await syntax:

- I don?t think we need ?twisted.deferredCoroutine? decorator ? that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ?asyncio.Task?, along with ?ensure_deferred? and/or ?create_task? APIs.  This way users can create Deferreds for coroutines only when they need it.

- ?ensure_deferred? could also accept instances of ?asyncio.Future?, and wrap them in a Deferred object.  This should make it possible to re-use asyncio-specific code in Twisted code.

Thoughts?

Thanks,
Yury

From njs at pobox.com  Fri Jun  3 18:26:19 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 3 Jun 2016 15:26:19 -0700
Subject: [Async-sig] A possible tiny step towards event loop interoperability
Message-ID: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>

Convincing everyone to agree on a single event loop API sounds hard.

But maybe it would be useful to start with a baby step of defining a
standard way for a coroutine to ask the event loop who it is, e.g.

  await what_event_loop_is_this()
    -> "asyncio"
    -> "curio"
    -> ...

?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From cory at lukasa.co.uk  Fri Jun  3 18:53:12 2016
From: cory at lukasa.co.uk (Cory Benfield)
Date: Fri, 3 Jun 2016 15:53:12 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
Message-ID: <A348DEDA-04F0-4827-8621-6E27F8EB92A2@lukasa.co.uk>


> On 3 Jun 2016, at 13:11, Yury Selivanov <yselivanov at gmail.com> wrote:
> 
> 2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ?asyncio.get_running_loop? function.  If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.
> 

Let?s pretend I?m an idiot for a moment. ;)

Why do we want Twisted to schedule callbacks in the asyncio style? Is this intended to be a compatibility mode for Twisted whereby in an asyncio runner Deferreds emulate Futures? Is it required that Deferreds behave in this way to be awaitable in Python 3.5+?

Cory
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/05e7ebd3/attachment.sig>

From cory at lukasa.co.uk  Fri Jun  3 18:54:13 2016
From: cory at lukasa.co.uk (Cory Benfield)
Date: Fri, 3 Jun 2016 15:54:13 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
Message-ID: <E2013963-85B3-41D1-B04F-22E89490D2DE@lukasa.co.uk>


> On 3 Jun 2016, at 15:26, Nathaniel Smith <njs at pobox.com> wrote:
> 
> Convincing everyone to agree on a single event loop API sounds hard.
> 
> But maybe it would be useful to start with a baby step of defining a
> standard way for a coroutine to ask the event loop who it is, e.g.
> 
>  await what_event_loop_is_this()
>    -> "asyncio"
>    -> "curio"
>    -> ...

How do we ensure that this function is always available in the global namespace, or alternatively that it?s always available in a *consistent* namespace?

Cory
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/766b9952/attachment.sig>

From njs at pobox.com  Fri Jun  3 19:21:13 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 3 Jun 2016 16:21:13 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <E2013963-85B3-41D1-B04F-22E89490D2DE@lukasa.co.uk>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <E2013963-85B3-41D1-B04F-22E89490D2DE@lukasa.co.uk>
Message-ID: <CAPJVwBkRsS4iwLcXEWN891jcPnMt+r0A0w7GrgjZi332UY8W_A@mail.gmail.com>

On Fri, Jun 3, 2016 at 3:54 PM, Cory Benfield <cory at lukasa.co.uk> wrote:
>
>> On 3 Jun 2016, at 15:26, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Convincing everyone to agree on a single event loop API sounds hard.
>>
>> But maybe it would be useful to start with a baby step of defining a
>> standard way for a coroutine to ask the event loop who it is, e.g.
>>
>>  await what_event_loop_is_this()
>>    -> "asyncio"
>>    -> "curio"
>>    -> ...
>
> How do we ensure that this function is always available in the global namespace, or alternatively that it?s always available in a *consistent* namespace?

It could go into asyncio, but also we could just document that the
function is literally

@types.coroutine
def what_event_loop_is_this():
    return (yield "WHAT-EVENT-LOOP-IS-THIS")

(actual value subject to bikeshedding -- the point is just that it's
some well-known value that the event loop can watch for and which
wouldn't be used for anything else.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From yselivanov at gmail.com  Fri Jun  3 19:37:11 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Fri, 3 Jun 2016 19:37:11 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
Message-ID: <057AE411-77FE-49AF-A564-5B209E156F38@gmail.com>

Given all the differences of let?s say curio vs asyncio, I don?t
see any real-world scenario where such function would be useful.

Yury

> On Jun 3, 2016, at 6:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> Convincing everyone to agree on a single event loop API sounds hard.
> 
> But maybe it would be useful to start with a baby step of defining a
> standard way for a coroutine to ask the event loop who it is, e.g.
> 
>  await what_event_loop_is_this()
>    -> "asyncio"
>    -> "curio"
>    -> ...
> 
> ?
> 
> -n
> 
> -- 
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/


From andrew.svetlov at gmail.com  Fri Jun  3 19:41:24 2016
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Fri, 03 Jun 2016 23:41:24 +0000
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwBkRsS4iwLcXEWN891jcPnMt+r0A0w7GrgjZi332UY8W_A@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <E2013963-85B3-41D1-B04F-22E89490D2DE@lukasa.co.uk>
 <CAPJVwBkRsS4iwLcXEWN891jcPnMt+r0A0w7GrgjZi332UY8W_A@mail.gmail.com>
Message-ID: <CAL3CFcXgk6FkK07EZTkFCWhqZjMWczQkp3AVw6Ot=t-WyYUobw@mail.gmail.com>

asyncio functions works with asyncio-compatible code only, isn't it?

For example asyncio.get_event_loop() should return a loop instance which
conforms asyncio.AbstractEventLoop ABC.

Third-party libraries like aiohttp just cannot work with arbitrary event
loop type. They are always require at least *compatible* loop instance.

On Fri, Jun 3, 2016 at 4:25 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Jun 3, 2016 at 3:54 PM, Cory Benfield <cory at lukasa.co.uk> wrote:
> >
> >> On 3 Jun 2016, at 15:26, Nathaniel Smith <njs at pobox.com> wrote:
> >>
> >> Convincing everyone to agree on a single event loop API sounds hard.
> >>
> >> But maybe it would be useful to start with a baby step of defining a
> >> standard way for a coroutine to ask the event loop who it is, e.g.
> >>
> >>  await what_event_loop_is_this()
> >>    -> "asyncio"
> >>    -> "curio"
> >>    -> ...
> >
> > How do we ensure that this function is always available in the global
> namespace, or alternatively that it?s always available in a *consistent*
> namespace?
>
> It could go into asyncio, but also we could just document that the
> function is literally
>
> @types.coroutine
> def what_event_loop_is_this():
>     return (yield "WHAT-EVENT-LOOP-IS-THIS")
>
> (actual value subject to bikeshedding -- the point is just that it's
> some well-known value that the event loop can watch for and which
> wouldn't be used for anything else.)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/

-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/6c37f981/attachment.html>

From yselivanov at gmail.com  Fri Jun  3 19:44:35 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Fri, 3 Jun 2016 19:44:35 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <A348DEDA-04F0-4827-8621-6E27F8EB92A2@lukasa.co.uk>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <A348DEDA-04F0-4827-8621-6E27F8EB92A2@lukasa.co.uk>
Message-ID: <840A4392-D563-4693-A5F1-34684C98E319@gmail.com>


> On Jun 3, 2016, at 6:53 PM, Cory Benfield <cory at lukasa.co.uk> wrote:
> 
> 
>> On 3 Jun 2016, at 13:11, Yury Selivanov <yselivanov at gmail.com> wrote:
>> 
>> 2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ?asyncio.get_running_loop? function.  If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.
>> 
> 
> Let?s pretend I?m an idiot for a moment. ;)
> 
> Why do we want Twisted to schedule callbacks in the asyncio style? Is this intended to be a compatibility mode for Twisted whereby in an asyncio runner Deferreds emulate Futures? Is it required that Deferreds behave in this way to be awaitable in Python 3.5+?


2.1 is not strictly necessary.  Although I would really prefer that any Twisted code I use within an asyncio application will behave like asyncio code (if that?s possible).

Yury

From guido at python.org  Fri Jun  3 20:07:12 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 3 Jun 2016 17:07:12 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
Message-ID: <CAP7+vJ+2u5a0H7vKKmn-mgSzY+mJgUUUnEKAkmVJHCuN-ui_QA@mail.gmail.com>

On Fri, Jun 3, 2016 at 3:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
> Convincing everyone to agree on a single event loop API sounds hard.

Really? That was exactly the purpose of asyncio and PEP 484 (and Glyph
helped a lot, so I kind of feel he has to some extent endorsed that
design).

-- 
--Guido van Rossum (python.org/~guido)

From ben at bendarnell.com  Fri Jun  3 20:14:22 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Fri, 3 Jun 2016 20:14:22 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
Message-ID: <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>

I think this could be useful, but's answering the question of "what
coroutine runner is this", not "what event loop is this". It's already
possible to stack one event loop on top of another so that the question of
event loop is ambiguous, but any `async def` function or other coroutine
has exactly one coroutine runner. The identity of the coroutine runner
matters more in practice than the event loop (PEP 484 has standardized the
event loop interface but left coroutine runners unspecified). For more on
this see https://github.com/tornadoweb/tornado/issues/1493 (in which
differences between coroutine runners became an issue in the context of the
event loop integration between Tornado and asyncio), and
https://github.com/tornadoweb/tornado/pull/1716 (in which asyncio.sleep
relies on certain behavior of the asyncio coroutine runner that prevents
its use in the tornado coroutine runner).

-Ben

On Fri, Jun 3, 2016 at 6:26 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Convincing everyone to agree on a single event loop API sounds hard.
>
> But maybe it would be useful to start with a baby step of defining a
> standard way for a coroutine to ask the event loop who it is, e.g.
>
>   await what_event_loop_is_this()
>     -> "asyncio"
>     -> "curio"
>     -> ...
>
> ?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/6fe73240/attachment-0001.html>

From njs at pobox.com  Fri Jun  3 20:25:41 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 3 Jun 2016 17:25:41 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAP7+vJ+2u5a0H7vKKmn-mgSzY+mJgUUUnEKAkmVJHCuN-ui_QA@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAP7+vJ+2u5a0H7vKKmn-mgSzY+mJgUUUnEKAkmVJHCuN-ui_QA@mail.gmail.com>
Message-ID: <CAPJVwBmPFLMuiSB0ViVMHb8w53CtJv-OFpJFEAkZ1RAx8fDk1Q@mail.gmail.com>

On Fri, Jun 3, 2016 at 5:07 PM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Jun 3, 2016 at 3:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> Convincing everyone to agree on a single event loop API sounds hard.
>
> Really? That was exactly the purpose of asyncio and PEP 484 (and Glyph
> helped a lot, so I kind of feel he has to some extent endorsed that
> design).

Sorry, that was partly referencing some discussion that happened at
PyCon before the mailing list existed... Tornado and Curio both
implement (partially) incompatible coroutine runners, and one of the
questions that came up was that people are worried about this (for
obvious reasons). I don't have a proposal for fixing this entirely,
but this proposal seemed like a simple step that might at least make
it possible in principle to write libraries that handled multiple
coroutine runners.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Fri Jun  3 20:26:15 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 3 Jun 2016 17:26:15 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
Message-ID: <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>

On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
> I think this could be useful, but's answering the question of "what
> coroutine runner is this", not "what event loop is this".

Thanks, that's definitely a better way to put it.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ben at bendarnell.com  Fri Jun  3 20:33:41 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Fri, 3 Jun 2016 20:33:41 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
Message-ID: <CAFkYKJ4E=SNXVHWwZi7gKOyKs-5yXtRo=JSyjPH8K6BNmyeu2w@mail.gmail.com>

On Fri, Jun 3, 2016 at 4:11 PM, Yury Selivanov <yselivanov at gmail.com> wrote:

> During the asyncio open space a few days ago we had a number of important
> questions raised, including Twisted/asyncio interop.  I?d like to start the
> discussion of a few related things:
>

For those who haven't seen it, it's already possible to mix asyncio,
twisted, and tornado freely in tornado coroutines (using the `yield`
syntax. `await` can be used with tornado and asyncio, but not twisted last
time I looked):
https://groups.google.com/forum/#!topic/python-tulip/W9_Cj4zN1Jc


>
> 1. key difference between a Deferred and a Future;
> 2. make Deferred objects compatible with asyncio code;
> 3. overall strategy to make Twisted code compatible with asyncio code.
>
> In detail:
>
> 1. Unlike Futures, Deferred objects know nothing about the event loop.
> Deferred calls its callbacks directly, while Future uses ?loop.call_soon?
> to ensure fair scheduling.
>

asyncio Futures know about their event loop, but concurrent futures and
Tornado futures do not. asyncio is the odd one out here, and I think it
would be better to make asyncio futures call their callbacks directly than
to introduce knowledge of the event loop to other deferred/future-like
objects.


>
> 2. I see two issues with Deferred/Future compatibility:
>
> 2.1. There should be a way for Deferred objects to get the currently
> running event loop reliably. There is a pending PR to add the new
> ?asyncio.get_running_loop? function.  If we land it in 3.5.2, Deferred
> object will be able to get the currently running event loop and schedule
> callbacks in a fair fashion, similarly to Futures.
>
> 2.2. asyncio uses ?isinstance(o, Future)? calls in many key places, such
> as ?asyncio.ensure_future? and in the Task class.  The latter makes it
> impossible for anything but Future and coroutines to be awaitable in
> asyncio code.  We have two options to fix this: make Future an instance of
> ABCMeta, or introduce a new protocol for Future-like objects.  The former
> option will not be considered because it makes isinstance calls much slower.
>

+1. This was my goal in the thread linked above.


>
> As for the second option ? a protocol for Future-like objects ? I propose
> to add a new dunder attribute ?__asyncio_future__ = True? to asyncio.Future
> class, and replace all ?isinstance(o, Future)? calls with ?hasattr(o,
> ?__asyncio_future__?)? checks.
>

Why make it asyncio specific? What you need is A) does it have an
add_done_callback() method with the right interface, and B) are callbacks
guaranteed to be run on the same thread (as with asyncio Futures) or is the
thread unspecified (as in concurrent Futures). The latter isn't really a
type-level check, so maybe Future instances should be associated with a
thread (rather than an event loop).


>
> (2.1) and (2.2) should make it possible for Twisted to make Deferred
> objects fully compatible with asyncio code.  Glyph and Amber, please
> confirm this.
>
> 3. I?ve been on Amber?s talk about Twisted and asyncio, and wanted to
> share my ideas on how Twisted should approach compatibility with asyncio
> and async/await syntax:
>
> - I don?t think we need ?twisted.deferredCoroutine? decorator ? that will
> make async/await a lot slower. Instead, Twisted should follow asyncio and
> create an alternative, tailored for Twisted, implementation of
> ?asyncio.Task?, along with ?ensure_deferred? and/or ?create_task? APIs.
> This way users can create Deferreds for coroutines only when they need it.


> - ?ensure_deferred? could also accept instances of ?asyncio.Future?, and
> wrap them in a Deferred object.  This should make it possible to re-use
> asyncio-specific code in Twisted code.
>

This is what Tornado's `convert_yielded` function does to allow twisted and
asyncio libraries to be used from tornado applications. (I can't remember
why we have to wrap asyncio futures in tornado futures; the interfaces are
compatible except for tornado's extensions to better capture stack traces
on python 2).

-Ben


>
> Thoughts?
>
> Thanks,
> Yury
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/3e8a991f/attachment.html>

From guido at python.org  Fri Jun  3 20:43:31 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 3 Jun 2016 17:43:31 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
Message-ID: <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>

Could someone point me to the specific code that's considered the
coroutine runner in asyncio and Tornado? I've been immersed in asyncio
for so long that I don't know which part you're talking about. :-(

On Fri, Jun 3, 2016 at 5:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
>> I think this could be useful, but's answering the question of "what
>> coroutine runner is this", not "what event loop is this".
>
> Thanks, that's definitely a better way to put it.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
--Guido van Rossum (python.org/~guido)

From ben at bendarnell.com  Fri Jun  3 20:45:39 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Fri, 3 Jun 2016 20:45:39 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
 <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>
Message-ID: <CAFkYKJ4ekAvqoqvU3xPOsX7gPTbN3Mzd4bxGt30D1ZD=MJcEtA@mail.gmail.com>

On Fri, Jun 3, 2016 at 8:43 PM, Guido van Rossum <guido at python.org> wrote:

> Could someone point me to the specific code that's considered the
> coroutine runner in asyncio and Tornado? I've been immersed in asyncio
> for so long that I don't know which part you're talking about. :-(
>

asyncio.Task and tornado.gen.Runner. Basically the thing that calls next()
and send() on generator objects.


>
> On Fri, Jun 3, 2016 at 5:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
> > On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
> >> I think this could be useful, but's answering the question of "what
> >> coroutine runner is this", not "what event loop is this".
> >
> > Thanks, that's definitely a better way to put it.
> >
> > -n
> >
> > --
> > Nathaniel J. Smith -- https://vorpus.org
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/759acfa6/attachment.html>

From lukasz at langa.pl  Fri Jun  3 20:46:55 2016
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 3 Jun 2016 17:46:55 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
Message-ID: <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>

Being able to re-use Twisted?s vast array of existing protocols and related libraries within asyncio would be a dream come true. Likewise, being able to write Twisted code using async/await coroutines without wrappers in-between would increase Twisted adoption since that syntax is more welcoming than callback-based Deferreds.

I have a few inline comments here.

> On Jun 3, 2016, at 1:11 PM, Yury Selivanov <yselivanov at gmail.com> wrote:
> 
> 2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ?asyncio.get_running_loop? function.  If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.

In general, writing polyglot code sucks. I always recommend sticking to a single version for applications. As library maintainers, we?re the poor people that have to provide compatibility across versions, typically with Tox.

I don?t know to what extent we can expect coroutines to be FULLY COMPATIBLE across async frameworks. They should be REUSABLE but that?s not the same thing. Specifically, having Deferreds calling their callbacks directly is perfectly fine by me. The goal here is to reuse what the Twisted ecosystem has already implemented so expecting all deferreds to now schedule callbacks on the event loop is not realistic. In the same vein, we can expect some coroutines to reuse asyncio internals (commingling with the event loop?s implementation details for instance) to the extent where compatibility with Twisted might not be possible.

Actually, +1 for Ben?s suggestion to revisit if asyncio.Futures really need call_soon scheduling of callbacks. Specifically, a torrent of call_soon doesn?t really achieve ?fair scheduling? as suggested by Yury anyway:

https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445

There?s still a comment on the asyncio codebase suggesting we might be able to merge concurrent.futures.Future with asyncio.Future. This would be a worthwhile goal.

> 2.2. asyncio uses ?isinstance(o, Future)? calls in many key places, such as ?asyncio.ensure_future? and in the Task class.  The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code.  We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects.  The former option will not be considered because it makes isinstance calls much slower.
> 
> As for the second option ? a protocol for Future-like objects ? I propose to add a new dunder attribute ?__asyncio_future__ = True? to asyncio.Future class, and replace all ?isinstance(o, Future)? calls with ?hasattr(o, ?__asyncio_future__?)? checks.
> 

+1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds.
+1 for Ben?s suggestion that we should avoid ?asyncio? in the name here. __isfuture__?


> - I don?t think we need ?twisted.deferredCoroutine? decorator ? that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ?asyncio.Task?, along with ?ensure_deferred? and/or ?create_task? APIs.  This way users can create Deferreds for coroutines only when they need it.

+1 here. Having to decorate async/await coroutines to be able to use them within Twisted is an adoption barrier. The same would be true about having to decorate Deferreds before awaiting on them in asyncio.

Amber, Glyph, Ben, is asyncio.Future?s event loop callback scheduling the biggest barrier in interoperability at the moment?

- ?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/b3d91e33/attachment.sig>

From guido at python.org  Fri Jun  3 20:48:49 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 3 Jun 2016 17:48:49 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAFkYKJ4ekAvqoqvU3xPOsX7gPTbN3Mzd4bxGt30D1ZD=MJcEtA@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
 <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>
 <CAFkYKJ4ekAvqoqvU3xPOsX7gPTbN3Mzd4bxGt30D1ZD=MJcEtA@mail.gmail.com>
Message-ID: <CAP7+vJJXd65-xAXZpau5QEHu7KJKHmSNViFiOUN6Dbb1YKtoUw@mail.gmail.com>

OK, that helps. Could you talk me through the crucial
incompatibilities here? (If that's too  much to ask I'm also happy to
leave this up to the experts -- I trust that there's more than enough
brainpower here to come up with a solution that works for all
concerned.)

On Fri, Jun 3, 2016 at 5:45 PM, Ben Darnell <ben at bendarnell.com> wrote:
> On Fri, Jun 3, 2016 at 8:43 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> Could someone point me to the specific code that's considered the
>> coroutine runner in asyncio and Tornado? I've been immersed in asyncio
>> for so long that I don't know which part you're talking about. :-(
>
>
> asyncio.Task and tornado.gen.Runner. Basically the thing that calls next()
> and send() on generator objects.
>
>>
>>
>> On Fri, Jun 3, 2016 at 5:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> > On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
>> >> I think this could be useful, but's answering the question of "what
>> >> coroutine runner is this", not "what event loop is this".
>> >
>> > Thanks, that's definitely a better way to put it.
>> >
>> > -n
>> >
>> > --
>> > Nathaniel J. Smith -- https://vorpus.org
>> > _______________________________________________
>> > Async-sig mailing list
>> > Async-sig at python.org
>> > https://mail.python.org/mailman/listinfo/async-sig
>> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>
>



-- 
--Guido van Rossum (python.org/~guido)

From ben at bendarnell.com  Fri Jun  3 21:10:30 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Fri, 3 Jun 2016 21:10:30 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAP7+vJJXd65-xAXZpau5QEHu7KJKHmSNViFiOUN6Dbb1YKtoUw@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
 <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>
 <CAFkYKJ4ekAvqoqvU3xPOsX7gPTbN3Mzd4bxGt30D1ZD=MJcEtA@mail.gmail.com>
 <CAP7+vJJXd65-xAXZpau5QEHu7KJKHmSNViFiOUN6Dbb1YKtoUw@mail.gmail.com>
Message-ID: <CAFkYKJ4nW2jVvX3h6dq7kAPbw5miJGauVbL0p-u9So8fShFsAA@mail.gmail.com>

On Fri, Jun 3, 2016 at 8:48 PM, Guido van Rossum <guido at python.org> wrote:

> OK, that helps. Could you talk me through the crucial
> incompatibilities here? (If that's too  much to ask I'm also happy to
> leave this up to the experts -- I trust that there's more than enough
> brainpower here to come up with a solution that works for all
> concerned.)
>

Coroutines that are chained together with either `yield from` or `await`
are essentially one big generator with one shared coroutine runner. At the
bottom, though, are the actual objects that are yielded by this generator
(in asyncio, this happens in `Future.__iter__`). If the coroutine runner is
asyncio.Task, it will raise if any part of this chain of coroutines yields
anything but None or an asyncio.Future. The Tornado coroutine runner is
more liberal: it accepts its own futures as well as asyncio and concurrent
futures or twisted deferreds.

In other words, this works with tornado, but not with asyncio:

import asyncio
import tornado.gen
import tornado.ioloop
import tornado.platform.asyncio

use_tornado = False

async def f():
    await asyncio.sleep(0.1)
    await tornado.gen.sleep(0.1)

if use_tornado:
    tornado.platform.asyncio.AsyncIOMainLoop().install()
    tornado.ioloop.IOLoop.current().run_sync(f)
else:
    asyncio.get_event_loop().run_until_complete(f())




There were more incompatibilities with the yield-based syntax than with the
await-based syntax. Tornado's coroutine runner allowed you to yield objects
like lists and dicts (yielding a container was a kind of fork/join) that
were not allowed in asyncio (thanks mainly to asyncio's use of `yield
from`). The introduction of `await` has imposed similar restrictions on
both systems, so there is no more yielding of lists or dicts in either
case; explicit wrapper objects must be used. (but there's still a
difference in that asyncio.Task allows None but tornado coroutines do not)





>
> On Fri, Jun 3, 2016 at 5:45 PM, Ben Darnell <ben at bendarnell.com> wrote:
> > On Fri, Jun 3, 2016 at 8:43 PM, Guido van Rossum <guido at python.org>
> wrote:
> >>
> >> Could someone point me to the specific code that's considered the
> >> coroutine runner in asyncio and Tornado? I've been immersed in asyncio
> >> for so long that I don't know which part you're talking about. :-(
> >
> >
> > asyncio.Task and tornado.gen.Runner. Basically the thing that calls
> next()
> > and send() on generator objects.
> >
> >>
> >>
> >> On Fri, Jun 3, 2016 at 5:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >> > On Fri, Jun 3, 2016 at 5:14 PM, Ben Darnell <ben at bendarnell.com>
> wrote:
> >> >> I think this could be useful, but's answering the question of "what
> >> >> coroutine runner is this", not "what event loop is this".
> >> >
> >> > Thanks, that's definitely a better way to put it.
> >> >
> >> > -n
> >> >
> >> > --
> >> > Nathaniel J. Smith -- https://vorpus.org
> >> > _______________________________________________
> >> > Async-sig mailing list
> >> > Async-sig at python.org
> >> > https://mail.python.org/mailman/listinfo/async-sig
> >> > Code of Conduct: https://www.python.org/psf/codeofconduct/
> >>
> >>
> >>
> >> --
> >> --Guido van Rossum (python.org/~guido)
> >
> >
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/94063284/attachment-0001.html>

From njs at pobox.com  Fri Jun  3 21:14:28 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 3 Jun 2016 18:14:28 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAFkYKJ4nW2jVvX3h6dq7kAPbw5miJGauVbL0p-u9So8fShFsAA@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
 <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>
 <CAFkYKJ4ekAvqoqvU3xPOsX7gPTbN3Mzd4bxGt30D1ZD=MJcEtA@mail.gmail.com>
 <CAP7+vJJXd65-xAXZpau5QEHu7KJKHmSNViFiOUN6Dbb1YKtoUw@mail.gmail.com>
 <CAFkYKJ4nW2jVvX3h6dq7kAPbw5miJGauVbL0p-u9So8fShFsAA@mail.gmail.com>
Message-ID: <CAPJVwB=icqLVnpvc6VsW4nD7OyFZAuvr6H7NX-ocg7k0xBu0Dg@mail.gmail.com>

On Fri, Jun 3, 2016 at 6:10 PM, Ben Darnell <ben at bendarnell.com> wrote:
> The introduction of `await` has imposed similar restrictions on both
> systems, so there is no more yielding of lists or dicts in either case;
> explicit wrapper objects must be used. (but there's still a difference in
> that asyncio.Task allows None but tornado coroutines do not)

I'm not saying that yielding lists or dicts is useful, but it sounds
like you're saying that somehow it's actually incompatible with
async/await, and I don't understand why that would be -- e.g., this
works fine?:

In [2]: @types.coroutine
   ...: def yield_list():
   ...:     yield [1, 2, 3]
   ...:

In [3]: async def f():
   ...:     await yield_list()
   ...:

In [4]: next(f().__await__())
Out[4]: [1, 2, 3]

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ben at bendarnell.com  Fri Jun  3 21:16:40 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Fri, 3 Jun 2016 21:16:40 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwB=icqLVnpvc6VsW4nD7OyFZAuvr6H7NX-ocg7k0xBu0Dg@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
 <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>
 <CAFkYKJ4ekAvqoqvU3xPOsX7gPTbN3Mzd4bxGt30D1ZD=MJcEtA@mail.gmail.com>
 <CAP7+vJJXd65-xAXZpau5QEHu7KJKHmSNViFiOUN6Dbb1YKtoUw@mail.gmail.com>
 <CAFkYKJ4nW2jVvX3h6dq7kAPbw5miJGauVbL0p-u9So8fShFsAA@mail.gmail.com>
 <CAPJVwB=icqLVnpvc6VsW4nD7OyFZAuvr6H7NX-ocg7k0xBu0Dg@mail.gmail.com>
Message-ID: <CAFkYKJ6D+9PaBn95afBKNBAEL9-2OzyiBa6VwtUcbg8AiG_itg@mail.gmail.com>

On Fri, Jun 3, 2016 at 9:14 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Jun 3, 2016 at 6:10 PM, Ben Darnell <ben at bendarnell.com> wrote:
> > The introduction of `await` has imposed similar restrictions on both
> > systems, so there is no more yielding of lists or dicts in either case;
> > explicit wrapper objects must be used. (but there's still a difference in
> > that asyncio.Task allows None but tornado coroutines do not)
>
> I'm not saying that yielding lists or dicts is useful, but it sounds
> like you're saying that somehow it's actually incompatible with
> async/await, and I don't understand why that would be -- e.g., this
> works fine?:
>

Yes, this works fine as long as you have the extra hop into a decorated
coroutine. What you can't do is `await [1, 2, 3]` in an `async def` native
coroutine.


>
> In [2]: @types.coroutine
>    ...: def yield_list():
>    ...:     yield [1, 2, 3]
>    ...:
>
> In [3]: async def f():
>    ...:     await yield_list()
>    ...:
>
> In [4]: next(f().__await__())
> Out[4]: [1, 2, 3]
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/962d78d2/attachment.html>

From njs at pobox.com  Fri Jun  3 21:33:54 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 3 Jun 2016 18:33:54 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAFkYKJ6D+9PaBn95afBKNBAEL9-2OzyiBa6VwtUcbg8AiG_itg@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <CAPJVwBnQVrF5MuEABTfnw2_GoBa6j+RP=EKL3xWFDk3jRVcehA@mail.gmail.com>
 <CAP7+vJKbtPL5-L0d+z94nXb1bXVOnmKK9XP2dLKQcs3TLuVPPQ@mail.gmail.com>
 <CAFkYKJ4ekAvqoqvU3xPOsX7gPTbN3Mzd4bxGt30D1ZD=MJcEtA@mail.gmail.com>
 <CAP7+vJJXd65-xAXZpau5QEHu7KJKHmSNViFiOUN6Dbb1YKtoUw@mail.gmail.com>
 <CAFkYKJ4nW2jVvX3h6dq7kAPbw5miJGauVbL0p-u9So8fShFsAA@mail.gmail.com>
 <CAPJVwB=icqLVnpvc6VsW4nD7OyFZAuvr6H7NX-ocg7k0xBu0Dg@mail.gmail.com>
 <CAFkYKJ6D+9PaBn95afBKNBAEL9-2OzyiBa6VwtUcbg8AiG_itg@mail.gmail.com>
Message-ID: <CAPJVwBketsw6sbvJCgGCderfX9dQx1fX=7fqDFSj4GNafk2RVA@mail.gmail.com>

On Fri, Jun 3, 2016 at 6:16 PM, Ben Darnell <ben at bendarnell.com> wrote:
> On Fri, Jun 3, 2016 at 9:14 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> On Fri, Jun 3, 2016 at 6:10 PM, Ben Darnell <ben at bendarnell.com> wrote:
>> > The introduction of `await` has imposed similar restrictions on both
>> > systems, so there is no more yielding of lists or dicts in either case;
>> > explicit wrapper objects must be used. (but there's still a difference
>> > in
>> > that asyncio.Task allows None but tornado coroutines do not)
>>
>> I'm not saying that yielding lists or dicts is useful, but it sounds
>> like you're saying that somehow it's actually incompatible with
>> async/await, and I don't understand why that would be -- e.g., this
>> works fine?:
>
>
> Yes, this works fine as long as you have the extra hop into a decorated
> coroutine. What you can't do is `await [1, 2, 3]` in an `async def` native
> coroutine.

Ahh, gotcha. So the constraint is that the final 'await' always has to
go through special awaitable object -- but there's no constraint on
what kinds of objects the leaf awaitable and the coroutine runner pass
back and forth to communicate.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From yselivanov at gmail.com  Fri Jun  3 22:17:34 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Fri, 3 Jun 2016 22:17:34 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
Message-ID: <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>

[..]
> Actually, +1 for Ben?s suggestion to revisit if asyncio.Futures really need call_soon scheduling of callbacks. Specifically, a torrent of call_soon doesn?t really achieve ?fair scheduling? as suggested by Yury anyway:
> 
> https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445

One quick story about call_soon: while working on uvloop I once spent a couple of hours debugging why asyncio/sslproto.py didn?t work.  Turned out that SSLProtocol, in its connection_made scheduled some callbacks with loop.call_soon; and data_received did that too.  Turned out that libuv called data_received one loop iteration earlier, which resulted in a race condition.  What I?m trying to say here, is that making a subtle difference in how callbacks are scheduled can introduce some hard to debug bugs.

In fact, I?m thinking now that Tornado/Twisted should keep their callbacks scheduling logic as is, and asyncio should keep its current implementation.


Anyways, I?ve tried to update asyncio Future to call callbacks directly:
https://github.com/1st1/asyncio/commit/9da7ff868f1405893947118178d62bb7e1eb3a51

The result is many broken unittests, until... a segfault in test_create_ssl_connection: https://gist.github.com/1st1/797dd12932bba7b353038fb7a80148ac (I?ll investigate this later).

I?d be -1 to change callbacks scheduling in 3.5, IMO this has to wait until 3.6 even if we decide this should be done.

[..]
> +1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds.
> +1 for Ben?s suggestion that we should avoid ?asyncio? in the name here. __isfuture__?

Yes, an asyncio-agnostic name would be better.  I like __isfuture__.


Yury

From yselivanov at gmail.com  Fri Jun  3 22:26:54 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Fri, 3 Jun 2016 22:26:54 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <CAFkYKJ4E=SNXVHWwZi7gKOyKs-5yXtRo=JSyjPH8K6BNmyeu2w@mail.gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <CAFkYKJ4E=SNXVHWwZi7gKOyKs-5yXtRo=JSyjPH8K6BNmyeu2w@mail.gmail.com>
Message-ID: <F9E69EBD-4855-47D0-B5C9-923E986FB458@gmail.com>

[..]
> 1. Unlike Futures, Deferred objects know nothing about the event loop.  Deferred calls its callbacks directly, while Future uses ?loop.call_soon? to ensure fair scheduling.
> 
> asyncio Futures know about their event loop, but concurrent futures and Tornado futures do not. asyncio is the odd one out here, and I think it would be better to make asyncio futures call their callbacks directly than to introduce knowledge of the event loop to other deferred/future-like objects.

Please see my reply to Lukasz.  In short there are some problems with changing how asyncio currently works.

[..]
> Why make it asyncio specific? What you need is A) does it have an add_done_callback() method with the right interface, and B) are callbacks guaranteed to be run on the same thread (as with asyncio Futures) or is the thread unspecified (as in concurrent Futures). The latter isn't really a type-level check, so maybe Future instances should be associated with a thread (rather than an event loop). 

Yes, (A) and (B) are both important, and they define a minimal set of requirements.  I?d say the Future should also be an awaitable, and support ?set_result? and other methods.

Associating Futures with threads is a neat idea, but maybe, a special attribute like __isfuture__ is the simplest option?

Yury


From yselivanov at gmail.com  Fri Jun  3 22:40:21 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Fri, 3 Jun 2016 22:40:21 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
Message-ID: <D4885922-72BC-4DE0-9E4D-8C62346D5420@gmail.com>


> On Jun 3, 2016, at 8:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
> 
> I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this". It's already possible to stack one event loop on top of another so that the question of event loop is ambiguous, but any `async def` function or other coroutine has exactly one coroutine runner. The identity of the coroutine runner matters more in practice than the event loop (PEP 484 has standardized the event loop interface but left coroutine runners unspecified).

Right, let?s discuss the hypothetical ?await what_coroutine_runner_is_this()?.  Which, in case of asyncio, would return an instance of the current Task, right?

> For more on this see https://github.com/tornadoweb/tornado/issues/1493 (in which differences between coroutine runners became an issue in the context of the event loop integration between Tornado and asyncio), and https://github.com/tornadoweb/tornado/pull/1716 (in which asyncio.sleep relies on certain behavior of the asyncio coroutine runner that prevents its use in the tornado coroutine runner). 

I?ve read through both of those issues, but I still don?t understand what problem (or how the linked problems) can be solved by getting the current coroutine runner.  Could you please provide an example?

Thanks,
Yury


From glyph at twistedmatrix.com  Fri Jun  3 22:53:05 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Fri, 3 Jun 2016 19:53:05 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAP7+vJ+2u5a0H7vKKmn-mgSzY+mJgUUUnEKAkmVJHCuN-ui_QA@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAP7+vJ+2u5a0H7vKKmn-mgSzY+mJgUUUnEKAkmVJHCuN-ui_QA@mail.gmail.com>
Message-ID: <5CA0D8AF-375E-42C0-AEBE-E3F02EAEF17E@twistedmatrix.com>


> On Jun 3, 2016, at 17:07, Guido van Rossum <guido at python.org> wrote:
> 
> On Fri, Jun 3, 2016 at 3:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> Convincing everyone to agree on a single event loop API sounds hard.
> 
> Really? That was exactly the purpose of asyncio and PEP 484 (and Glyph
> helped a lot, so I kind of feel he has to some extent endorsed that
> design).

To echo that, the asyncio loop is a pretty solid basis for cross-framework compatibility.  It hasn't been completely zero-friction for Twisted to adopt it, but work is ongoing there and I don't think there are any blockers, aside from "time to make it happen".  But as has already come up on this thread, we're really talking about the coroutine scheduler.

Specifically where this came up in the openspace was in the ability to await a Deferred.  And "what event loop is this" is not entirely what we want to await, it is "give me the event loop".

Here's the problem: if an asyncio coroutine wants to await upon a Deferred, the Deferred does not know about the event loop.  But when a Deferred is awaited, it must answer two questions: (A), does it need to convert itself into a Future, or is the caller natively expecting a Deferred, and (B) if it needs to create a Future, what 'loop' should it be passing to said Future?

The reason the distinction is important is that if the caller natively expects a Deferred, then the callback chain can be invoked synchronously, whereas if the caller expects a Future, the loop needs to be made available for call_soon.  And it is the loop into which said Future will eventually be awaited that determines which one should be passed in.

For what it's worth, if Future didn't depend on the loop at all, but received the loop as an argument somehow along the awaitability chain, this issue would disappear (at least for this specific case).

Is that explanation clear?

-glyph


From glyph at twistedmatrix.com  Fri Jun  3 22:58:29 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Fri, 3 Jun 2016 19:58:29 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
Message-ID: <F49540E0-E86A-4505-90B0-67DC4555EE2D@twistedmatrix.com>


> On Jun 3, 2016, at 19:17, Yury Selivanov <yselivanov at gmail.com> wrote:
> 
> In fact, I?m thinking now that Tornado/Twisted should keep their callbacks scheduling logic as is, and asyncio should keep its current implementation.

I have to agree.  There are are good reasons for both the asyncio and the Twisted ways of doing things, and they have subtly different priorities that cascade out into pretty much every application; changing it might cause things to break in more or less arbitrary ways.

>> +1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds.
>> +1 for Ben?s suggestion that we should avoid ?asyncio? in the name here. __isfuture__?
> 
> Yes, an asyncio-agnostic name would be better.  I like __isfuture__.

__futuristic__? :)

-glyph


From njs at pobox.com  Fri Jun  3 23:29:12 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 3 Jun 2016 20:29:12 -0700
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <D4885922-72BC-4DE0-9E4D-8C62346D5420@gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <D4885922-72BC-4DE0-9E4D-8C62346D5420@gmail.com>
Message-ID: <CAPJVwBmJvLeDo=ORA2wqR9v9tNmQ41YV=zn7MWyswvp8WWyWbA@mail.gmail.com>

On Fri, Jun 3, 2016 at 7:40 PM, Yury Selivanov <yselivanov at gmail.com> wrote:
>
>> On Jun 3, 2016, at 8:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
>>
>> I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this". It's already possible to stack one event loop on top of another so that the question of event loop is ambiguous, but any `async def` function or other coroutine has exactly one coroutine runner. The identity of the coroutine runner matters more in practice than the event loop (PEP 484 has standardized the event loop interface but left coroutine runners unspecified).
>
> Right, let?s discuss the hypothetical ?await what_coroutine_runner_is_this()?.  Which, in case of asyncio, would return an instance of the current Task, right?

I guess what I was really thinking of was

  await what_api_does_this_coroutine_runner_implement()

so if you really needed to you could write code that does
asyncio-things on asyncio, curio-things on curio, etc., and the caller
doesn't have to care.

I'll emphasize again that this is a trivial tiny suggestion :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From yselivanov at gmail.com  Fri Jun  3 23:34:20 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Fri, 3 Jun 2016 23:34:20 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <5CA0D8AF-375E-42C0-AEBE-E3F02EAEF17E@twistedmatrix.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAP7+vJ+2u5a0H7vKKmn-mgSzY+mJgUUUnEKAkmVJHCuN-ui_QA@mail.gmail.com>
 <5CA0D8AF-375E-42C0-AEBE-E3F02EAEF17E@twistedmatrix.com>
Message-ID: <644BECD8-95FB-4F47-A945-90D424DB8A2B@gmail.com>


> On Jun 3, 2016, at 10:53 PM, Glyph <glyph at twistedmatrix.com> wrote:
> 
> 
>> On Jun 3, 2016, at 17:07, Guido van Rossum <guido at python.org> wrote:
>> 
>> On Fri, Jun 3, 2016 at 3:26 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>> Convincing everyone to agree on a single event loop API sounds hard.
>> 
>> Really? That was exactly the purpose of asyncio and PEP 484 (and Glyph
>> helped a lot, so I kind of feel he has to some extent endorsed that
>> design).
> 
> To echo that, the asyncio loop is a pretty solid basis for cross-framework compatibility.  It hasn't been completely zero-friction for Twisted to adopt it, but work is ongoing there and I don't think there are any blockers, aside from "time to make it happen".  But as has already come up on this thread, we're really talking about the coroutine scheduler.
> 
> Specifically where this came up in the openspace was in the ability to await a Deferred.  And "what event loop is this" is not entirely what we want to await, it is "give me the event loop".
> 
> Here's the problem: if an asyncio coroutine wants to await upon a Deferred, the Deferred does not know about the event loop.  But when a Deferred is awaited, it must answer two questions: (A), does it need to convert itself into a Future, or is the caller natively expecting a Deferred, and (B) if it needs to create a Future, what 'loop' should it be passing to said Future?


If we:

1. agree that asyncio.Future will continue to use call_soon for callbacks, and Deferreds will continue to just call them;

2. we replace isinstance(Future) checks with __isfuture__ attribute; 

3. Deferred adds __isfuture__, implements __await__(), add_done_callback() and cancel() methods;

then:

- Coroutines run with asyncio.Task will be able to await on Deferreds (created by Twisted code);

- Coroutines run with Twisted?s coroutine runner will be able to await on Futures (created by asyncio code).

Is that correct?  Or we need something else?

Thanks,
Yury

From yselivanov at gmail.com  Sat Jun  4 00:08:09 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Sat, 4 Jun 2016 00:08:09 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <CAPJVwBmJvLeDo=ORA2wqR9v9tNmQ41YV=zn7MWyswvp8WWyWbA@mail.gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <D4885922-72BC-4DE0-9E4D-8C62346D5420@gmail.com>
 <CAPJVwBmJvLeDo=ORA2wqR9v9tNmQ41YV=zn7MWyswvp8WWyWbA@mail.gmail.com>
Message-ID: <C54A4D7B-1DC4-4175-BA48-8ED27BAB192E@gmail.com>


> On Jun 3, 2016, at 11:29 PM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> On Fri, Jun 3, 2016 at 7:40 PM, Yury Selivanov <yselivanov at gmail.com> wrote:
>> 
>>> On Jun 3, 2016, at 8:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
>>> 
>>> I think this could be useful, but's answering the question of "what coroutine runner is this", not "what event loop is this". It's already possible to stack one event loop on top of another so that the question of event loop is ambiguous, but any `async def` function or other coroutine has exactly one coroutine runner. The identity of the coroutine runner matters more in practice than the event loop (PEP 484 has standardized the event loop interface but left coroutine runners unspecified).
>> 
>> Right, let?s discuss the hypothetical ?await what_coroutine_runner_is_this()?.  Which, in case of asyncio, would return an instance of the current Task, right?
> 
> I guess what I was really thinking of was
> 
>  await what_api_does_this_coroutine_runner_implement()
> 
> so if you really needed to you could write code that does
> asyncio-things on asyncio, curio-things on curio, etc., and the caller
> doesn't have to care.
> 
> I'll emphasize again that this is a trivial tiny suggestion :-)


Glyph and Ben want information about the coroutine runner in __await__, where awaiting on something is not possible.

If we are talking about coroutines that users of curio/twisted/tornado/asyncio will write, then I don?t think they will need that thing either ? it?s too low level.

What makes interoperability between asyncio, tornado and twisted possible is that they all use similar event loops designs.  They all have something similar to call_soon and other callbacks related APIs, Future/Deferred etc.  What?s good about Futures and Deferreds is that they are well-defined objects with APIs etc.

In curio, David intentionally didn?t want to have anything similar to Futures and Deferreds.  Instead, he introduced a concept of ?traps? (see [1]), which are generators wrapped with ?@types.coroutine? yielding hardcoded tuples, that curio ?kernel? can understand.  Which is a very neat and clever design, except that it doesn?t compose with anything else.  With that design, it?s essentially impossible for asyncio coroutines to await on curio coroutines. The only way would be to implement all of curio APIs in asyncio in a wrapper to do this:

   async def asyncio_coroutine():
        await curio_adapter(curio_coroutine())

I?m not sure if it?s possible for curio coroutines to transparently await on asyncio coroutines without a significant refactoring of curio.  But even if David decides to do that, for compatibility with Futures he?ll only need to add ?isinstance(trap, Future)? in his kernel code, no new APIs are needed (IMO).

[1] https://github.com/dabeaz/curio/blob/master/curio/traps.py#L25

Yury


From ben at bendarnell.com  Sat Jun  4 16:25:43 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Sat, 4 Jun 2016 16:25:43 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
Message-ID: <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>

On Fri, Jun 3, 2016 at 10:17 PM, Yury Selivanov <yselivanov at gmail.com>
wrote:

> [..]
> > Actually, +1 for Ben?s suggestion to revisit if asyncio.Futures really
> need call_soon scheduling of callbacks. Specifically, a torrent of
> call_soon doesn?t really achieve ?fair scheduling? as suggested by Yury
> anyway:
> >
> > https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445
>
> One quick story about call_soon: while working on uvloop I once spent a
> couple of hours debugging why asyncio/sslproto.py didn?t work.  Turned out
> that SSLProtocol, in its connection_made scheduled some callbacks with
> loop.call_soon; and data_received did that too.  Turned out that libuv
> called data_received one loop iteration earlier, which resulted in a race
> condition.  What I?m trying to say here, is that making a subtle difference
> in how callbacks are scheduled can introduce some hard to debug bugs.
>

If things are so sensitive to minor changes in timing, doesn't that set the
bar impossibly high for interoperability? In Tornado, we have similar
concerns around "fair scheduling" of coroutines, but we solved this by
using our equivalent of call_soon() in the coroutine runner rather than in
the Future itself. So asyncio Futures yielded in a coroutine that uses
Tornado's runner will be called one iteration later as they make two trips
through the scheduler, while if asyncio.Task gained the ability to
understand Tornado Futures they could turn into a busy loop that let
nothing else run. If we want interoperability, we're going to have to
mandate that code like SSLProtocol be tolerant of changes in timing (of
course, saying that is one thing and achieving it is another)


>
> In fact, I?m thinking now that Tornado/Twisted should keep their callbacks
> scheduling logic as is, and asyncio should keep its current implementation.
>

Yeah, the risks of making any changes in this area outweigh the benefits,
since we'll never get all the implementations to schedule things in exactly
the same way.

-Ben


>
>
> Anyways, I?ve tried to update asyncio Future to call callbacks directly:
>
> https://github.com/1st1/asyncio/commit/9da7ff868f1405893947118178d62bb7e1eb3a51
>
> The result is many broken unittests, until... a segfault in
> test_create_ssl_connection:
> https://gist.github.com/1st1/797dd12932bba7b353038fb7a80148ac (I?ll
> investigate this later).
>
> I?d be -1 to change callbacks scheduling in 3.5, IMO this has to wait
> until 3.6 even if we decide this should be done.
>
> [..]
> > +1 for replacing isinstance checks with an attribute that Twisted can
> add to Deferreds.
> > +1 for Ben?s suggestion that we should avoid ?asyncio? in the name here.
> __isfuture__?
>
> Yes, an asyncio-agnostic name would be better.  I like __isfuture__.
>
>
> Yury
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160604/72704490/attachment.html>

From ben at bendarnell.com  Sat Jun  4 17:06:05 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Sat, 4 Jun 2016 17:06:05 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
Message-ID: <CAFkYKJ47u_emZsO_v8Cy3z7KBbFYMCeM5u-xnJSczNY2OnA5WA@mail.gmail.com>

On Fri, Jun 3, 2016 at 8:46 PM, ?ukasz Langa <lukasz at langa.pl> wrote:

>
> Amber, Glyph, Ben, is asyncio.Future?s event loop callback scheduling the
> biggest barrier in interoperability at the moment?
>

 For Tornado, this was not an issue. Things are perhaps not as efficient as
they could be (and this is why tornado.concurrent.Future cannot be an alias
for asyncio.Future), but it basically just works.

The biggest problem that arises when people use Tornado and asyncio
together is that some asyncio functions (such as asyncio.gather()) are
written to accept bare coroutines and run them in asyncio.Task. This works
in a pure asyncio world, but fails in an application that tries to mix the
two frameworks. Since Tornado's coroutine runner is essentially a superset
of asyncio's, the workaround is simple: use the Tornado version of any
functions that exhibit this problem (e.g. tornado.gen.multi() instead of
asyncio.gather()). And while I'm happy to promote Tornado's more flexible
coroutine runner over asyncio's, it's clearly problematic for
standardization and interop. This approach only works so long as there is
some coroutine runner that is a superset of all the others.

The root of the problem is that without a standardized coroutine runner,
async functions are difficult to use: each asynchronous function embeds
assumptions about what runner will be used, and these assumptions will need
to be documented along with instructions for what to do when you need to
cross runner boundaries. We kind of have a standardized coroutine runner in
async.Task, but it's inflexible, dealing only in asyncio.Futures. I think
the right thing to do is to give asyncio a functools.singledispatch-based
hook like the one in Tornado (tornado.gen.convert_yielded) for converting
yielded objects into Futures. If we could register Tornado's Futures with
asyncio then I don't think there would be a reason to prefer Tornado's
coroutine runner when asyncio is available.

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160604/9e3c6caf/attachment.html>

From ben at bendarnell.com  Sat Jun  4 17:11:00 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Sat, 4 Jun 2016 17:11:00 -0400
Subject: [Async-sig] A possible tiny step towards event loop
 interoperability
In-Reply-To: <D4885922-72BC-4DE0-9E4D-8C62346D5420@gmail.com>
References: <CAPJVwBkw8jwhocyU_558hDMO_yizxObTWDXkda2O6Wi8a3nH_A@mail.gmail.com>
 <CAFkYKJ5hi6ZwP+DNPrivAJfmiD2-vB3fRZciYqvH6MEuGP5+KQ@mail.gmail.com>
 <D4885922-72BC-4DE0-9E4D-8C62346D5420@gmail.com>
Message-ID: <CAFkYKJ5-1oMhZ5tZ+5V9wBiV95ypB7v_Zfnim0A4E_PYbDv1Aw@mail.gmail.com>

On Fri, Jun 3, 2016 at 10:40 PM, Yury Selivanov <yselivanov at gmail.com>
wrote:

>
> > On Jun 3, 2016, at 8:14 PM, Ben Darnell <ben at bendarnell.com> wrote:
> >
> > I think this could be useful, but's answering the question of "what
> coroutine runner is this", not "what event loop is this". It's already
> possible to stack one event loop on top of another so that the question of
> event loop is ambiguous, but any `async def` function or other coroutine
> has exactly one coroutine runner. The identity of the coroutine runner
> matters more in practice than the event loop (PEP 484 has standardized the
> event loop interface but left coroutine runners unspecified).
>
> Right, let?s discuss the hypothetical ?await
> what_coroutine_runner_is_this()?.  Which, in case of asyncio, would return
> an instance of the current Task, right?
>

I was thinking it would just return a string like "asyncio", but returning
the Task itself could be useful too.


>
> > For more on this see https://github.com/tornadoweb/tornado/issues/1493
> (in which differences between coroutine runners became an issue in the
> context of the event loop integration between Tornado and asyncio), and
> https://github.com/tornadoweb/tornado/pull/1716 (in which asyncio.sleep
> relies on certain behavior of the asyncio coroutine runner that prevents
> its use in the tornado coroutine runner).
>
> I?ve read through both of those issues, but I still don?t understand what
> problem (or how the linked problems) can be solved by getting the current
> coroutine runner.  Could you please provide an example?
>
>
Neither of these are concrete issues that would be solved by exposing the
coroutine runner, but they're an example of how the coroutine runner often
matters more to the application than the event loop implementation (which
you can already get with type(asyncio.get_event_loop())). A better example
is the difference between tornado.gen.multi and asyncio.gather (
https://github.com/tornadoweb/tornado/issues/1684). If the coroutine runner
is asyncio, then asyncio.gather must be used and tornado.gen.multi won't
work. If the coroutine runner is Tornado, then tornado.gen.multi is
preferred; asyncio.gather will not work if any of the coroutines passed to
it assume the tornado coroutine runner.

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160604/dec4a569/attachment-0001.html>

From brett at python.org  Sat Jun  4 12:17:00 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 04 Jun 2016 16:17:00 +0000
Subject: [Async-sig] Inviting people to this list
Message-ID: <CAP1=2W50=bvNi0h4uejiv6=Ub6JATspuRt3=f1YR4+DyfWr+bw@mail.gmail.com>

We have folks from asyncio, h2/h11, tornado, and Twisted. David Beazley
knows about the list but he told me he isn't a mailing list guy so we will
see if he joins. There is no one from gevent that I know of, so if someone
thinks they should be here then please invite them to join. If there are
other specific projects that you think should have someone here then please
reach out to them.

Otherwise I'm personally not going out of my way to broadcast this list as
I'm happy with who's here and the level of discourse (although no need to
hide the list's existence either). ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160604/aa5583ff/attachment.html>

From ben at bendarnell.com  Sun Jun  5 11:56:30 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Sun, 5 Jun 2016 11:56:30 -0400
Subject: [Async-sig] asyncio.timeout() is not portable
Message-ID: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>

It has come to my attention (https://github.com/KeepSafe/aiohttp/issues/877)
that Python 3.5.2 (!) introduced a new context manager asyncio.timeout,
which attaches a timeout to the current asyncio.Task. Any library or
application that uses this feature will not be portable to other coroutine
runners.

    with asyncio.timeout(1):
        result = await aiohttp.get('http://www.example.com')

It's difficult to make this interface portable. We'd need to introduce a
global thread-specific object that all of the coroutine runners could
register the current task on, and define an interface that asyncio.timeout
would call on that object to set the timeout. We could get rid of the
global if the context manager used `async with` instead of `with`, since
that would give us a way to communicate with the coroutine runner directly,
but the net result is still that each runner needs to implement hooks for
this feature (the hooks may or may not be generalizable to other features
in the future).

If our goal is portability across coroutine runners, then asyncio.timeout()
needs to go back to the drawing board. If the goal is to extend
asyncio.Task to be the one true coroutine runner then we could leave it
as-is.

For comparison, Tornado's equivalent to asyncio.timeout is
tornado.gen.with_timeout(), and it has no special interactions with the
coroutine runner:

    result = await with_timeout(timedelta(seconds=1),
                     convert_yielded(aiohttp.get("http://www.example.com")))

(The convert_yielded call won't be needed in the next release of Tornado.
The use of timedelta is because Tornado prefers absolute deadlines instead
of relative timeouts, so that's how plain numbers are interpreted).

-Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160605/ebf0359d/attachment.html>

From njs at pobox.com  Sun Jun  5 12:40:25 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 5 Jun 2016 09:40:25 -0700
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
Message-ID: <CAPJVwBkLmHu254G8ToXJAoPnQZqqggC4Yc3dr94px6uKS00QUw@mail.gmail.com>

On Jun 5, 2016 8:57 AM, "Ben Darnell" <ben at bendarnell.com> wrote:
>
> It has come to my attention (
https://github.com/KeepSafe/aiohttp/issues/877) that Python 3.5.2 (!)
introduced a new context manager asyncio.timeout, which attaches a timeout
to the current asyncio.Task. Any library or application that uses this
feature will not be portable to other coroutine runners.
>
>     with asyncio.timeout(1):
>         result = await aiohttp.get('http://www.example.com')
>
> It's difficult to make this interface portable. We'd need to introduce a
global thread-specific object that all of the coroutine runners could
register the current task on, and define an interface that asyncio.timeout
would call on that object to set the timeout. We could get rid of the
global if the context manager used `async with` instead of `with`, since
that would give us a way to communicate with the coroutine runner directly,
but the net result is still that each runner needs to implement hooks for
this feature (the hooks may or may not be generalizable to other features
in the future).

Right -- fwiw curio's version of this uses either 'async with' (like you
suggest) or acts as a wrapper (like tornado):
    http://curio.readthedocs.io/en/latest/reference.html#timeouts

But this is consistent with these library's respective conventions:
asyncio's style in general is to look up the event loop using global state
that has to be kept synchronized, and here that's extended to looking up
the current coroutine runner as well. Curio, OTOH, never manipulates global
state. So to me this points to one of the fundamental differences between
these approaches; it doesn't seem like something where one can just look at
the timeout function in isolation.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160605/20e11aeb/attachment.html>

From guido at python.org  Sun Jun  5 13:02:31 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 5 Jun 2016 10:02:31 -0700
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
Message-ID: <CAP7+vJLnx4yYKBmmQGknVBMTLPgMCngzXNuihj9LXMiHUzRboQ@mail.gmail.com>

So in the case of that Tornado example, who does the sleeping (or how
is the exception scheduled)? We may not be able to retract this from
3.5.2 (RC1 goes out next weekend) but asyncio is still provisional
until the 3.6 feature freeze in September. What's a better design?
(FWIW asyncio.timeout() takes an optional loop parameter, but that's
probably not helpful.)

On Sun, Jun 5, 2016 at 8:56 AM, Ben Darnell <ben at bendarnell.com> wrote:
> It has come to my attention (https://github.com/KeepSafe/aiohttp/issues/877)
> that Python 3.5.2 (!) introduced a new context manager asyncio.timeout,
> which attaches a timeout to the current asyncio.Task. Any library or
> application that uses this feature will not be portable to other coroutine
> runners.
>
>     with asyncio.timeout(1):
>         result = await aiohttp.get('http://www.example.com')
>
> It's difficult to make this interface portable. We'd need to introduce a
> global thread-specific object that all of the coroutine runners could
> register the current task on, and define an interface that asyncio.timeout
> would call on that object to set the timeout. We could get rid of the global
> if the context manager used `async with` instead of `with`, since that would
> give us a way to communicate with the coroutine runner directly, but the net
> result is still that each runner needs to implement hooks for this feature
> (the hooks may or may not be generalizable to other features in the future).
>
> If our goal is portability across coroutine runners, then asyncio.timeout()
> needs to go back to the drawing board. If the goal is to extend asyncio.Task
> to be the one true coroutine runner then we could leave it as-is.
>
> For comparison, Tornado's equivalent to asyncio.timeout is
> tornado.gen.with_timeout(), and it has no special interactions with the
> coroutine runner:
>
>     result = await with_timeout(timedelta(seconds=1),
>                      convert_yielded(aiohttp.get("http://www.example.com")))
>
> (The convert_yielded call won't be needed in the next release of Tornado.
> The use of timedelta is because Tornado prefers absolute deadlines instead
> of relative timeouts, so that's how plain numbers are interpreted).
>
> -Ben
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/



-- 
--Guido van Rossum (python.org/~guido)

From ben at bendarnell.com  Sun Jun  5 13:24:19 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Sun, 5 Jun 2016 13:24:19 -0400
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAP7+vJLnx4yYKBmmQGknVBMTLPgMCngzXNuihj9LXMiHUzRboQ@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <CAP7+vJLnx4yYKBmmQGknVBMTLPgMCngzXNuihj9LXMiHUzRboQ@mail.gmail.com>
Message-ID: <CAFkYKJ5ci9mgWPCsv=H1yHOF9H386ewh0kybQF8nYjMT1MzW9g@mail.gmail.com>

On Sun, Jun 5, 2016 at 1:02 PM, Guido van Rossum <guido at python.org> wrote:

> So in the case of that Tornado example, who does the sleeping (or how
> is the exception scheduled)?


The exception is scheduled with IOLoop.add_timeout (equivalent to asyncio's
call_later). One difference between the two is that Tornado's with_timeout
creates a new future (and a new coroutine runner) instead of cancelling the
input future.

The code is here:
https://github.com/tornadoweb/tornado/blob/11bd951cc04747c0f03c71575d004f322950a5c3/tornado/gen.py#L835


> We may not be able to retract this from
> 3.5.2 (RC1 goes out next weekend) but asyncio is still provisional
> until the 3.6 feature freeze in September. What's a better design?
>

I prefer a function that wraps an awaitable (as in Tornado's
implementation), but this does have a performance impact. If we want to
avoid that, then I think an `async with` context manager would be a better
choice than an ordinary context manager, but I'd like to have a proof of
concept implementation that works across asyncio and at least one other
platform before committing to that design.



> (FWIW asyncio.timeout() takes an optional loop parameter, but that's
> probably not helpful.)
>

No, it's not helpful because event loops and coroutine runners are
independent. It's easy to find the current event loop (there are globals
for that), but you can't really interact with the coroutine runner except
via yield/await (unless we introduce new globals for this purpose)

-Ben


>
> On Sun, Jun 5, 2016 at 8:56 AM, Ben Darnell <ben at bendarnell.com> wrote:
> > It has come to my attention (
> https://github.com/KeepSafe/aiohttp/issues/877)
> > that Python 3.5.2 (!) introduced a new context manager asyncio.timeout,
> > which attaches a timeout to the current asyncio.Task. Any library or
> > application that uses this feature will not be portable to other
> coroutine
> > runners.
> >
> >     with asyncio.timeout(1):
> >         result = await aiohttp.get('http://www.example.com')
> >
> > It's difficult to make this interface portable. We'd need to introduce a
> > global thread-specific object that all of the coroutine runners could
> > register the current task on, and define an interface that
> asyncio.timeout
> > would call on that object to set the timeout. We could get rid of the
> global
> > if the context manager used `async with` instead of `with`, since that
> would
> > give us a way to communicate with the coroutine runner directly, but the
> net
> > result is still that each runner needs to implement hooks for this
> feature
> > (the hooks may or may not be generalizable to other features in the
> future).
> >
> > If our goal is portability across coroutine runners, then
> asyncio.timeout()
> > needs to go back to the drawing board. If the goal is to extend
> asyncio.Task
> > to be the one true coroutine runner then we could leave it as-is.
> >
> > For comparison, Tornado's equivalent to asyncio.timeout is
> > tornado.gen.with_timeout(), and it has no special interactions with the
> > coroutine runner:
> >
> >     result = await with_timeout(timedelta(seconds=1),
> >                      convert_yielded(aiohttp.get("http://www.example.com
> ")))
> >
> > (The convert_yielded call won't be needed in the next release of Tornado.
> > The use of timedelta is because Tornado prefers absolute deadlines
> instead
> > of relative timeouts, so that's how plain numbers are interpreted).
> >
> > -Ben
> >
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160605/d3240423/attachment-0001.html>

From njs at pobox.com  Sun Jun  5 14:38:29 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 5 Jun 2016 11:38:29 -0700
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAFkYKJ5ci9mgWPCsv=H1yHOF9H386ewh0kybQF8nYjMT1MzW9g@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <CAP7+vJLnx4yYKBmmQGknVBMTLPgMCngzXNuihj9LXMiHUzRboQ@mail.gmail.com>
 <CAFkYKJ5ci9mgWPCsv=H1yHOF9H386ewh0kybQF8nYjMT1MzW9g@mail.gmail.com>
Message-ID: <CAPJVwBmL4WC1-0ip7zdK0o3tbK2FkwzWxWd=N-Y9TpXU3Pe=yg@mail.gmail.com>

On Sun, Jun 5, 2016 at 10:24 AM, Ben Darnell <ben at bendarnell.com> wrote:
[...]
> No, it's not helpful because event loops and coroutine runners are
> independent. It's easy to find the current event loop (there are globals for
> that), but you can't really interact with the coroutine runner except via
> yield/await (unless we introduce new globals for this purpose)

Of course the reason it takes the loop argument is that in asyncio,
event loops and coroutine runners aren't independent at all -- the
loop is a global, and then the current coroutine runner is part of the
loop state, so effectively the current coroutine runner is already a
global:
    https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.current_task

In the curio approach they also aren't independent -- they don't have
any magic globals, but for them the event loop and coroutine runner
are the same object.

And AFAICT the tornado.gen.with_timeout wrapper doesn't deal with
coroutines/coroutine runners at all? I just read the source, but I
couldn't fully follow it. If I write
  async def do_stuff():
      try:
          await aiohttp.get("https://python.org")
      except:
          print("oops, cleaning up")
          raise
      else:
          print("did it!")
  await tornado.gen.with_timeout(timedelta(seconds=0.0001), do_stuff())
and the timeout fires, then is the output "oops, cleaning up", "did
it!", or nothing at all?

I guess the main case where coroutine runners are really independent
from an event loop is when you have random little ad hoc coroutine
runners that aren't directly integrated with an event loop at all.
E.g., in my async_generator package the @async_generator decorator
acts as a coroutine runner that adapts a regular coroutine into an
async iterator, by intercepting special 'await yield_(...)' messages
and transparently proxying other messages to the underlying coroutine
runner. Before I read the source I was expecting that maybe
tornado.gen.timeout_after would work in a broadly similar way, but
apparently I was wrong :-).

I don't have any argument here -- just trying to think aloud to
understand the design space better...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From andrew.svetlov at gmail.com  Sun Jun  5 17:12:45 2016
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sun, 05 Jun 2016 21:12:45 +0000
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAPJVwBmL4WC1-0ip7zdK0o3tbK2FkwzWxWd=N-Y9TpXU3Pe=yg@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <CAP7+vJLnx4yYKBmmQGknVBMTLPgMCngzXNuihj9LXMiHUzRboQ@mail.gmail.com>
 <CAFkYKJ5ci9mgWPCsv=H1yHOF9H386ewh0kybQF8nYjMT1MzW9g@mail.gmail.com>
 <CAPJVwBmL4WC1-0ip7zdK0o3tbK2FkwzWxWd=N-Y9TpXU3Pe=yg@mail.gmail.com>
Message-ID: <CAL3CFcUqnWEkEvw+xVvX=4fpYDo8P1w2JOq7WsCaGtUhLNsnVQ@mail.gmail.com>

Regarding to asyncio event loop globalness.

In my code I never use implicit global event loop but always pass the
instance everywhere when needed.

aiohttp as well as other aio-libs libraries supports optional loop=None
parameter.
The only reason why the loop is optional -- we are trying to follow asyncio
conventions.
But frankly speaking I found implicit event loop both non-obvious and
error-prone.

It would be nice to always pass event loop explicitly in asyncio-based code
but ship has sailed years ago.

On Sun, Jun 5, 2016 at 11:38 AM Nathaniel Smith <njs at pobox.com> wrote:

> On Sun, Jun 5, 2016 at 10:24 AM, Ben Darnell <ben at bendarnell.com> wrote:
> [...]
> > No, it's not helpful because event loops and coroutine runners are
> > independent. It's easy to find the current event loop (there are globals
> for
> > that), but you can't really interact with the coroutine runner except via
> > yield/await (unless we introduce new globals for this purpose)
>
> Of course the reason it takes the loop argument is that in asyncio,
> event loops and coroutine runners aren't independent at all -- the
> loop is a global, and then the current coroutine runner is part of the
> loop state, so effectively the current coroutine runner is already a
> global:
>
> https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.current_task
>
> In the curio approach they also aren't independent -- they don't have
> any magic globals, but for them the event loop and coroutine runner
> are the same object.
>
> And AFAICT the tornado.gen.with_timeout wrapper doesn't deal with
> coroutines/coroutine runners at all? I just read the source, but I
> couldn't fully follow it. If I write
>   async def do_stuff():
>       try:
>           await aiohttp.get("https://python.org")
>       except:
>           print("oops, cleaning up")
>           raise
>       else:
>           print("did it!")
>   await tornado.gen.with_timeout(timedelta(seconds=0.0001), do_stuff())
> and the timeout fires, then is the output "oops, cleaning up", "did
> it!", or nothing at all?
>
> I guess the main case where coroutine runners are really independent
> from an event loop is when you have random little ad hoc coroutine
> runners that aren't directly integrated with an event loop at all.
> E.g., in my async_generator package the @async_generator decorator
> acts as a coroutine runner that adapts a regular coroutine into an
> async iterator, by intercepting special 'await yield_(...)' messages
> and transparently proxying other messages to the underlying coroutine
> runner. Before I read the source I was expecting that maybe
> tornado.gen.timeout_after would work in a broadly similar way, but
> apparently I was wrong :-).
>
> I don't have any argument here -- just trying to think aloud to
> understand the design space better...
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160605/9fd897f5/attachment.html>

From glyph at twistedmatrix.com  Mon Jun  6 01:16:37 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Sun, 5 Jun 2016 22:16:37 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
Message-ID: <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>


> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
> 
> If things are so sensitive to minor changes in timing, doesn't that set the bar impossibly high for interoperability? 

The sensitivity is not to changes in timing - i.e. when the wall-clock runs, or ordering of non-deterministically ordered events - but rather to reentrancy - whether certain things complete synchronously while the caller is still on the stack and can depend on them having done so upon return.

The recommended way of writing tests within Twisted these days depends heavily on `.callback? synchronously resolving a Deferred, which is what adding a call_soon breaks.

-glyph


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160605/25f0e0ff/attachment.html>

From christoph at grothesque.org  Mon Jun  6 08:38:19 2016
From: christoph at grothesque.org (Christoph Groth)
Date: Mon, 06 Jun 2016 14:38:19 +0200
Subject: [Async-sig] async executors
Message-ID: <87wpm2332c.fsf@grothesque.org>

Hi,

A few days ago I posted a message [1] on python-ideas to which no 
one replied except Brett who suggested re-posting it here, which I 
do now.  The following is a complete rewrite/update of the 
original message.  I promise: I will stop the spamming if no one 
replies this time!

Async programming with executors seems to be a useful approach for 
number-crunching problems where the main algorithm (the one 
launching the futures) is quite complicated and possibly 
concurrent, but light enough to run on a single core (the heavy 
lifting is done ?inside? the futures).

This use case is addressed by asyncio's run_in_executor that uses 
the executors provided by the concurrent.futures package.  There 
is, however, a lot of duplication of functionality between 
concurrent.futures and asyncio.  (The main difference is that 
concurrent.futures relies on threading and locks.)  If one is 
going to use concurrent.futures only from coroutines that run 
under the control of the asyncio event loop, it should be possible 
to replace the concurrent.futures executors by async versions of 
them.

Why could such an async executor be interesting?

? It can be a lot simpler (and thus easier to understand and 
  extend) because it builds on top of asyncio.  There have been 
  already discussions on unifying the Future type from 
  concurrent.futures with that of asyncio [2].

? Performance is better, especially when many worker processes are 
  involved, because of the simpler code that uses less locking 
  (the only locking that remains is inside the multiprocessing 
  module).

As far as I can see the original 
concurrent.futures.ProcessPoolExecutor has no advantage when used 
in asyncio-based programs except when some coroutine blocks for 
long enough for the call queue to become empty.  (But hey, async 
programming is about non-blocking coroutines!)

Are there other possible advantages or disadvantages?

Based on concurrent.futures.ProcessPoolExecutor, I?ve made a 
proof-of-concept implementation [3] that demonstrates that the 
idea works.  (There have been some improvements compared to the 
version that I posted on python-ideas.)

Christoph


PS.

I would be grateful if any asyncio experts could have a look at 
the part of the main loop of the process management coroutine 
where the coroutine awaits new data [4].  Currently, I am using an 
asyncio.Event that is set by a callback via asyncio?s 
add_reader().  Is this the most natural way to do it currently? 
I?m particularly puzzled by the fact that reader.poll(0) is not 
always true after the event has been set.

If there is no better way to combine asyncio and 
multiprocessing.Process, perhaps this is a thing that could be 
improved in a future version of the Python stdlib?


[1] http://thread.gmane.org/gmane.comp.python.ideas/40709
[2] 
http://thread.gmane.org/gmane.comp.python.ideas/24551/focus=24557
[3] https://gitlab.kwant-project.org/cwg/aexecutor
[4] 
https://gitlab.kwant-project.org/cwg/aexecutor/blob/9290504e779c543d9ee93adcb1de36390a69eaad/aexecutor/process.py#L298
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/5e88d6b4/attachment-0001.sig>

From guido at python.org  Mon Jun  6 11:29:01 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 6 Jun 2016 08:29:01 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
 <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
Message-ID: <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>

On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com> wrote:
>
> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
>
> If things are so sensitive to minor changes in timing, doesn't that set the
> bar impossibly high for interoperability?
>
>
> The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
> or ordering of non-deterministically ordered events - but rather to
> reentrancy - whether certain things complete synchronously while the caller
> is still on the stack and can depend on them having done so upon return.
>
> The recommended way of writing tests within Twisted these days depends
> heavily on `.callback? synchronously resolving a Deferred, which is what
> adding a call_soon breaks.

That's interesting, and also potentially worrisome (for interop, I'm
not saying Twisted is wrong here).

I think asyncio depends on the opposite: that if you add a callback to
a Future that's ready it does *not* immediately run. Asyncio's promise
is pretty strongly that callbacks are serialized (no callbacks running
inside other callbacks). IIRC we experimented with other semantics and
found that it was harder to reason about. (IMO if you *know* a Future
is ready why add a callback to it rather than just calling the damn
thing if that's what you want?)

-- 
--Guido van Rossum (python.org/~guido)

From njs at pobox.com  Mon Jun  6 12:29:37 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 6 Jun 2016 09:29:37 -0700
Subject: [Async-sig] async executors
In-Reply-To: <CAPJVwBkEtb_-Q4w1LfW9Qzqw6aD0Fvu0dWJBtpgDkks2jiT99A@mail.gmail.com>
References: <87wpm2332c.fsf@grothesque.org>
 <CAPJVwBkjObY0HxXFekVby-1zBDZZTPZBSxnwDz3C5xDi98Eo3w@mail.gmail.com>
 <CAPJVwBkEtb_-Q4w1LfW9Qzqw6aD0Fvu0dWJBtpgDkks2jiT99A@mail.gmail.com>
Message-ID: <CAPJVwBmPJnwT4X--hAO6B3ytOH7jOZPxVNhN8rrp0CPx_6AUeg@mail.gmail.com>

On Jun 6, 2016 5:37 AM, "Christoph Groth" <christoph at grothesque.org> wrote:
>
[...]
> Async programming with executors seems to be a useful approach for
number-crunching problems where the main algorithm (the one launching the
futures) is quite complicated and possibly concurrent, but light enough to
run on a single core (the heavy lifting is done ?inside? the futures).
>
> This use case is addressed by asyncio's run_in_executor that uses the
executors provided by the concurrent.futures package.  There is, however, a
lot of duplication of functionality between concurrent.futures and
asyncio.  (The main difference is that concurrent.futures relies on
threading and locks.)  If one is going to use concurrent.futures only from
coroutines that run under the control of the asyncio event loop, it should
be possible to replace the concurrent.futures executors by async versions
of them.

Sounds like an interesting project :-)

FYI, the latest versions of curio also have their own thread/process pool
implementations that replace concurrent.futures, I think for similar
reasons to the ones you allude to:

http://curio.readthedocs.io/en/latest/reference.html#performing-external-work

There are certainly reasons you might prefer to use asyncio instead of
curio, but if you're implementing such a thing on asyncio then curio's
implementation and semantics might be interesting for reference.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/799dce30/attachment.html>

From yselivanov at gmail.com  Mon Jun  6 12:52:51 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Mon, 6 Jun 2016 12:52:51 -0400
Subject: [Async-sig] async executors
In-Reply-To: <87wpm2332c.fsf@grothesque.org>
References: <87wpm2332c.fsf@grothesque.org>
Message-ID: <DB5F2662-C11E-4700-900C-4991FB400387@gmail.com>

Hi Christoph,

[..]
> ? Performance is better, especially when many worker processes are  involved, because of the simpler code that uses less locking  (the only locking that remains is inside the multiprocessing  module).

Yes, it will be better, but the performance improvements will be visible only for 100s or 1000s of processes.  And only if they are executing short-living tasks.

Is there a real need for a bit faster process pool?

[..]
> Are there other possible advantages or disadvantages?

1. Backwards compatibility?  We?ll have to continue to support concurrent.futures executors.

2. More code in asyncio core -> more work for alternative event loop implementations such as uvloop.

3. Stability.  concurrent.futures is mature and well tested.

> Based on concurrent.futures.ProcessPoolExecutor, I?ve made a proof-of-concept implementation [3] that demonstrates that the idea works.  (There have been some improvements compared to the version that I posted on python-ideas.)

I think you should stabilize the code base, and release it as a module on PyPI.

[..]
> I would be grateful if any asyncio experts could have a look at the part of the main loop of the process management coroutine where the coroutine awaits new data [4].  Currently, I am using an asyncio.Event that is set by a callback via asyncio?s add_reader().  Is this the most natural way to do it currently? 

I think you can use asyncio.Future for resuming that coroutine.

Thanks,
Yury


From yselivanov at gmail.com  Mon Jun  6 12:55:45 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Mon, 6 Jun 2016 12:55:45 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <CAFkYKJ47u_emZsO_v8Cy3z7KBbFYMCeM5u-xnJSczNY2OnA5WA@mail.gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <CAFkYKJ47u_emZsO_v8Cy3z7KBbFYMCeM5u-xnJSczNY2OnA5WA@mail.gmail.com>
Message-ID: <E3DC261F-3005-4F9C-BA12-4B615863304B@gmail.com>


> On Jun 4, 2016, at 5:06 PM, Ben Darnell <ben at bendarnell.com> wrote:
> 
> The root of the problem is that without a standardized coroutine runner, async functions are difficult to use: each asynchronous function embeds assumptions about what runner will be used, and these assumptions will need to be documented along with instructions for what to do when you need to cross runner boundaries. We kind of have a standardized coroutine runner in async.Task, but it's inflexible, dealing only in asyncio.Futures. I think the right thing to do is to give asyncio a functools.singledispatch-based hook like the one in Tornado (tornado.gen.convert_yielded) for converting yielded objects into Futures. If we could register Tornado's Futures with asyncio then I don't think there would be a reason to prefer Tornado's coroutine runner when asyncio is available.

Looks like you, Glyph and I should work on a PEP to standardize asyncio.Task and make it more flexible.

I can draft a first version (in a couple of weeks if that works), that will basically describe the current Task implementation, but I?ll need your help to make it useful for Tornado and Twisted.

Thanks,
Yury

From yselivanov at gmail.com  Mon Jun  6 13:15:51 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Mon, 6 Jun 2016 13:15:51 -0400
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
Message-ID: <EB1F6EE7-4A2F-4C41-AA79-E31338A7166B@gmail.com>


> On Jun 5, 2016, at 11:56 AM, Ben Darnell <ben at bendarnell.com> wrote:
> 
> It has come to my attention (https://github.com/KeepSafe/aiohttp/issues/877) that Python 3.5.2 (!) introduced a new context manager asyncio.timeout, which attaches a timeout to the current asyncio.Task. Any library or application that uses this feature will not be portable to other coroutine runners.
> 
>     with asyncio.timeout(1):
>         result = await aiohttp.get('http://www.example.com')
> 
> It's difficult to make this interface portable. We'd need to introduce a global thread-specific object that all of the coroutine runners could register the current task on, and define an interface that asyncio.timeout would call on that object to set the timeout. We could get rid of the global if the context manager used `async with` instead of `with`, since that would give us a way to communicate with the coroutine runner directly, but the net result is still that each runner needs to implement hooks for this feature (the hooks may or may not be generalizable to other features in the future).

We?ve discussed this and decided that we should revert asyncio.Timeout from 3.5.2.  I?ll do that later today.

Yury

From lukasz at langa.pl  Mon Jun  6 14:32:58 2016
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Mon, 6 Jun 2016 11:32:58 -0700
Subject: [Async-sig] async executors
In-Reply-To: <87wpm2332c.fsf@grothesque.org>
References: <87wpm2332c.fsf@grothesque.org>
Message-ID: <041210F2-1DC3-4543-83E8-F51DE8E807A4@langa.pl>


> On Jun 6, 2016, at 5:38 AM, Christoph Groth <christoph at grothesque.org> wrote:
> 
> As far as I can see the original concurrent.futures.ProcessPoolExecutor has no advantage when used in asyncio-based programs except when some coroutine blocks for long enough for the call queue to become empty.  (But hey, async programming is about non-blocking coroutines!)

That?s incorrect. The ProcessPoolExecutor is a way for you to run CPU-intensive tasks without affecting the responsiveness of the main event loop in your program. As long as you keep your arguments small (pickling/unpickling big data structures is pretty detrimental to performance here) and the duration of the task is non-trivial, the child processes enable you to use more CPU cores, and since they have their own GILs, this doesn't affect your loop at all, unlike using a thread pool.

I agree that the process pool is useless for a torrent of very fine-grained tasks where locking, futures and IPC are a significant fraction of the work to be done. But that doesn?t mean the process pool executor has no advantage. We?re using the PPE just fine at Facebook in cases where using TPE was being affected by the GIL.

--
Lukasz Langa | Facebook
Production Engineer | The Ministry of Silly Walks
(+1) 650-681-7811
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/9d4446e5/attachment-0001.sig>

From njs at pobox.com  Mon Jun  6 15:37:29 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 6 Jun 2016 12:37:29 -0700
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAPJVwB=_2OanSA4=BWBihGbFWdpdOq3q+pc0atEOWjxNjF9L_Q@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <EB1F6EE7-4A2F-4C41-AA79-E31338A7166B@gmail.com>
 <CAPJVwBnj4E2dmRQ+EGWGBCTSR+xeJ7ogAPfob2_N61jfEntVKg@mail.gmail.com>
 <CAPJVwB=_2OanSA4=BWBihGbFWdpdOq3q+pc0atEOWjxNjF9L_Q@mail.gmail.com>
Message-ID: <CAPJVwB=K64OJposYkx5SqvGuB3DERmU0upvao-th_c6TU9Ys_A@mail.gmail.com>

On Jun 6, 2016 10:19 AM, "Yury Selivanov" <yselivanov at gmail.com> wrote:
>
>
> > On Jun 5, 2016, at 11:56 AM, Ben Darnell <ben at bendarnell.com> wrote:
> >
> > It has come to my attention (
https://github.com/KeepSafe/aiohttp/issues/877) that Python 3.5.2 (!)
introduced a new context manager asyncio.timeout, which attaches a timeout
to the current asyncio.Task. Any library or application that uses this
feature will not be portable to other coroutine runners.
> >
> >     with asyncio.timeout(1):
> >         result = await aiohttp.get('http://www.example.com')
> >
> > It's difficult to make this interface portable. We'd need to introduce
a global thread-specific object that all of the coroutine runners could
register the current task on, and define an interface that asyncio.timeout
would call on that object to set the timeout. We could get rid of the
global if the context manager used `async with` instead of `with`, since
that would give us a way to communicate with the coroutine runner directly,
but the net result is still that each runner needs to implement hooks for
this feature (the hooks may or may not be generalizable to other features
in the future).
>
> We?ve discussed this and decided that we should revert asyncio.Timeout
from 3.5.2.  I?ll do that later today.

This is tangential and somewhat irrelevant given the above, but since I'm
not sure where else to put it: having thought about this issue a bit more
last night, I also think that it's important that the timeout manager
(if/when it's reintroduced) should inject a meaningful exception like
TimeoutError, and it should be legal for the coroutine to catch this and
continue. This is in contrast to the one that's currently being reverted,
which uses the task cancellation feature and is indistinguishable from any
other use of the task cancellation feature.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/85810851/attachment.html>

From glyph at twistedmatrix.com  Mon Jun  6 16:54:53 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Mon, 6 Jun 2016 13:54:53 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
 <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
 <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>
Message-ID: <2F7C1C4A-39F4-4A58-AA25-96E61C20DCCB@twistedmatrix.com>


> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org> wrote:
> 
> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com> wrote:
>> 
>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
>> 
>> If things are so sensitive to minor changes in timing, doesn't that set the
>> bar impossibly high for interoperability?
>> 
>> 
>> The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
>> or ordering of non-deterministically ordered events - but rather to
>> reentrancy - whether certain things complete synchronously while the caller
>> is still on the stack and can depend on them having done so upon return.
>> 
>> The recommended way of writing tests within Twisted these days depends
>> heavily on `.callback? synchronously resolving a Deferred, which is what
>> adding a call_soon breaks.
> 
> That's interesting, and also potentially worrisome (for interop, I'm
> not saying Twisted is wrong here).
> 
> I think asyncio depends on the opposite: that if you add a callback to
> a Future that's ready it does *not* immediately run. Asyncio's promise
> is pretty strongly that callbacks are serialized (no callbacks running
> inside other callbacks). IIRC we experimented with other semantics and
> found that it was harder to reason about. (IMO if you *know* a Future
> is ready why add a callback to it rather than just calling the damn
> thing if that's what you want?)

I don't think Twisted is necessarily right here either.  You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred.  I like the property where you can do:

def test(test_case):
    a = asynchronously_something()
    test_case.this_must_not_have_a_result(a)
    cause_a_to_fire()
    test_case.assertEqual(test_case.this_must_have_a_result(a), something)

but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does.  So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without.  You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/c581999c/attachment.html>

From guido at python.org  Mon Jun  6 17:21:54 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 6 Jun 2016 14:21:54 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <2F7C1C4A-39F4-4A58-AA25-96E61C20DCCB@twistedmatrix.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
 <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
 <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>
 <2F7C1C4A-39F4-4A58-AA25-96E61C20DCCB@twistedmatrix.com>
Message-ID: <CAP7+vJJtzswz+LE1qgS5TJX=UxafB=VOiZ_GwXRzBY02-FaiOg@mail.gmail.com>

On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com> wrote:

>
>
> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org> wrote:
>
> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com> wrote:
>
>
> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
>
> If things are so sensitive to minor changes in timing, doesn't that set the
> bar impossibly high for interoperability?
>
>
> The sensitivity is not to changes in timing - i.e. when the wall-clock
> runs,
> or ordering of non-deterministically ordered events - but rather to
> reentrancy - whether certain things complete synchronously while the caller
> is still on the stack and can depend on them having done so upon return.
>
> The recommended way of writing tests within Twisted these days depends
> heavily on `.callback? synchronously resolving a Deferred, which is what
> adding a call_soon breaks.
>
>
> That's interesting, and also potentially worrisome (for interop, I'm
> not saying Twisted is wrong here).
>
> I think asyncio depends on the opposite: that if you add a callback to
> a Future that's ready it does *not* immediately run. Asyncio's promise
> is pretty strongly that callbacks are serialized (no callbacks running
> inside other callbacks). IIRC we experimented with other semantics and
> found that it was harder to reason about. (IMO if you *know* a Future
> is ready why add a callback to it rather than just calling the damn
> thing if that's what you want?)
>
>
> I don't think Twisted is necessarily *right* here either.  You're
> absolutely right that it's easier to reason about reentrancy in some cases
> if you have a might-be-fired-might-not Future vs. the same sort of
> Deferred.  I like the property where you can do:
>
> def test(test_case):
>     a = asynchronously_something()
>     test_case.this_must_not_have_a_result(a)
>     cause_a_to_fire()
>     test_case.assertEqual(test_case.this_must_have_a_result(a), something)
>
> but this is (somewhat) opposed to the fact that call_soon means you never
> get the nasty surprise where the callback added in the middle of a function
> gets run before the rest of it does.  So I think there are good properties
> in both cases and given some thought it is probably possible to map between
> them, but Deferred is lower-level here in the sense that it provides a way
> to do this both with the event loop and without.  You can always
> call_soon(deferred.callback) but you can't go the other way and force a
> Future to resolve synchronously - right?
>

Right. I'm still unclear on what the compelling use case for that is (other
than that Twisted has always done this). Is it performance? Is it callback
ordering?

I suppose Deferred has a method to mark it done. Does that immediately run
the callbacks? Or does it come in two flavors? Can a Deferred that's marked
done ever revert back to being not done? (I believe I once read the
Deferred code enough to be able to find the answers, but I'm afraid I've
never really needed what I learned then, so I've forgotten...)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/a9124b30/attachment-0001.html>

From glyph at twistedmatrix.com  Mon Jun  6 17:35:50 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Mon, 6 Jun 2016 14:35:50 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <CAP7+vJJtzswz+LE1qgS5TJX=UxafB=VOiZ_GwXRzBY02-FaiOg@mail.gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
 <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
 <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>
 <2F7C1C4A-39F4-4A58-AA25-96E61C20DCCB@twistedmatrix.com>
 <CAP7+vJJtzswz+LE1qgS5TJX=UxafB=VOiZ_GwXRzBY02-FaiOg@mail.gmail.com>
Message-ID: <5BD7621E-5D57-4AB4-9A90-647124ED4F56@twistedmatrix.com>


> On Jun 6, 2016, at 14:21, Guido van Rossum <guido at python.org> wrote:
> 
> On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> 
> 
>> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote:
>> 
>> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
>>> 
>>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com <mailto:ben at bendarnell.com>> wrote:
>>> 
>>> If things are so sensitive to minor changes in timing, doesn't that set the
>>> bar impossibly high for interoperability?
>>> 
>>> 
>>> The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
>>> or ordering of non-deterministically ordered events - but rather to
>>> reentrancy - whether certain things complete synchronously while the caller
>>> is still on the stack and can depend on them having done so upon return.
>>> 
>>> The recommended way of writing tests within Twisted these days depends
>>> heavily on `.callback? synchronously resolving a Deferred, which is what
>>> adding a call_soon breaks.
>> 
>> That's interesting, and also potentially worrisome (for interop, I'm
>> not saying Twisted is wrong here).
>> 
>> I think asyncio depends on the opposite: that if you add a callback to
>> a Future that's ready it does *not* immediately run. Asyncio's promise
>> is pretty strongly that callbacks are serialized (no callbacks running
>> inside other callbacks). IIRC we experimented with other semantics and
>> found that it was harder to reason about. (IMO if you *know* a Future
>> is ready why add a callback to it rather than just calling the damn
>> thing if that's what you want?)
> 
> I don't think Twisted is necessarily right here either.  You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred.  I like the property where you can do:
> 
> def test(test_case):
>     a = asynchronously_something()
>     test_case.this_must_not_have_a_result(a)
>     cause_a_to_fire()
>     test_case.assertEqual(test_case.this_must_have_a_result(a), something)
> 
> but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does.  So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without.  You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
> 
> Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?

Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it.  So the main use-case is testing.

> I suppose Deferred has a method to mark it done.

Yep; ".callback".

> Does that immediately run the callbacks?

It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.

There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.

So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.

> Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?

No.  A Deferred that has been called back stays called back; callbacking it again is always an error.  However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.

> (I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)

Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160606/ed93171a/attachment.html>

From andrew.svetlov at gmail.com  Wed Jun  8 13:15:18 2016
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Wed, 08 Jun 2016 17:15:18 +0000
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAPJVwB=K64OJposYkx5SqvGuB3DERmU0upvao-th_c6TU9Ys_A@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <EB1F6EE7-4A2F-4C41-AA79-E31338A7166B@gmail.com>
 <CAPJVwBnj4E2dmRQ+EGWGBCTSR+xeJ7ogAPfob2_N61jfEntVKg@mail.gmail.com>
 <CAPJVwB=_2OanSA4=BWBihGbFWdpdOq3q+pc0atEOWjxNjF9L_Q@mail.gmail.com>
 <CAPJVwB=K64OJposYkx5SqvGuB3DERmU0upvao-th_c6TU9Ys_A@mail.gmail.com>
Message-ID: <CAL3CFcWwC96WWa2B46kOR8U_UZSj7Vyi6Aw7FxPtHZHC_GT7WA@mail.gmail.com>

asyncio.timeout behaves like asyncio.wait_for BTW.
It cancels inner tasks and sends asyncio.TimeoutError to outer.


On Mon, Jun 6, 2016 at 12:37 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Jun 6, 2016 10:19 AM, "Yury Selivanov" <yselivanov at gmail.com> wrote:
> >
> >
> > > On Jun 5, 2016, at 11:56 AM, Ben Darnell <ben at bendarnell.com> wrote:
> > >
> > > It has come to my attention (
> https://github.com/KeepSafe/aiohttp/issues/877) that Python 3.5.2 (!)
> introduced a new context manager asyncio.timeout, which attaches a timeout
> to the current asyncio.Task. Any library or application that uses this
> feature will not be portable to other coroutine runners.
> > >
> > >     with asyncio.timeout(1):
> > >         result = await aiohttp.get('http://www.example.com')
> > >
> > > It's difficult to make this interface portable. We'd need to introduce
> a global thread-specific object that all of the coroutine runners could
> register the current task on, and define an interface that asyncio.timeout
> would call on that object to set the timeout. We could get rid of the
> global if the context manager used `async with` instead of `with`, since
> that would give us a way to communicate with the coroutine runner directly,
> but the net result is still that each runner needs to implement hooks for
> this feature (the hooks may or may not be generalizable to other features
> in the future).
> >
> > We?ve discussed this and decided that we should revert asyncio.Timeout
> from 3.5.2.  I?ll do that later today.
>
> This is tangential and somewhat irrelevant given the above, but since I'm
> not sure where else to put it: having thought about this issue a bit more
> last night, I also think that it's important that the timeout manager
> (if/when it's reintroduced) should inject a meaningful exception like
> TimeoutError, and it should be legal for the coroutine to catch this and
> continue. This is in contrast to the one that's currently being reverted,
> which uses the task cancellation feature and is indistinguishable from any
> other use of the task cancellation feature.
>
> -n
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/

-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160608/e8165482/attachment.html>

From yselivanov at gmail.com  Wed Jun  8 13:49:29 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Wed, 8 Jun 2016 13:49:29 -0400
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAL3CFcWwC96WWa2B46kOR8U_UZSj7Vyi6Aw7FxPtHZHC_GT7WA@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <EB1F6EE7-4A2F-4C41-AA79-E31338A7166B@gmail.com>
 <CAPJVwBnj4E2dmRQ+EGWGBCTSR+xeJ7ogAPfob2_N61jfEntVKg@mail.gmail.com>
 <CAPJVwB=_2OanSA4=BWBihGbFWdpdOq3q+pc0atEOWjxNjF9L_Q@mail.gmail.com>
 <CAPJVwB=K64OJposYkx5SqvGuB3DERmU0upvao-th_c6TU9Ys_A@mail.gmail.com>
 <CAL3CFcWwC96WWa2B46kOR8U_UZSj7Vyi6Aw7FxPtHZHC_GT7WA@mail.gmail.com>
Message-ID: <37D25E18-A67E-4639-ABE8-9A9BF468753E@gmail.com>


> On Jun 8, 2016, at 1:15 PM, Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
> 
> asyncio.timeout behaves like asyncio.wait_for BTW.
> It cancels inner tasks and sends asyncio.TimeoutError to outer.

Right.  And I think it?s a totally normal behaviour for asyncio.

The only thing I?d fix in asyncio.timeout() (aside from resolving
Ben?s questions) is to make it use ?async with? statement.

Yury


From yselivanov at gmail.com  Wed Jun  8 13:52:45 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Wed, 8 Jun 2016 13:52:45 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <5BD7621E-5D57-4AB4-9A90-647124ED4F56@twistedmatrix.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
 <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
 <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>
 <2F7C1C4A-39F4-4A58-AA25-96E61C20DCCB@twistedmatrix.com>
 <CAP7+vJJtzswz+LE1qgS5TJX=UxafB=VOiZ_GwXRzBY02-FaiOg@mail.gmail.com>
 <5BD7621E-5D57-4AB4-9A90-647124ED4F56@twistedmatrix.com>
Message-ID: <36BA2A9F-9F14-4CE6-976F-17689C405C09@gmail.com>

Glyph, Ben, Amber,

So what?s the resolution on Future.__isfuture__ and fixing the
isinstance(obj, Future) checks from asyncio?

3.5.2 RC is only few days away, I can still make the change if
it?s a blocker for Twisted and Tornado.

Yury

> On Jun 6, 2016, at 5:35 PM, Glyph <glyph at twistedmatrix.com> wrote:
> 
>> 
>> On Jun 6, 2016, at 14:21, Guido van Rossum <guido at python.org> wrote:
>> 
>> On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com> wrote:
>> 
>> 
>>> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org> wrote:
>>> 
>>> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com> wrote:
>>>> 
>>>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
>>>> 
>>>> If things are so sensitive to minor changes in timing, doesn't that set the
>>>> bar impossibly high for interoperability?
>>>> 
>>>> 
>>>> The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
>>>> or ordering of non-deterministically ordered events - but rather to
>>>> reentrancy - whether certain things complete synchronously while the caller
>>>> is still on the stack and can depend on them having done so upon return.
>>>> 
>>>> The recommended way of writing tests within Twisted these days depends
>>>> heavily on `.callback? synchronously resolving a Deferred, which is what
>>>> adding a call_soon breaks.
>>> 
>>> That's interesting, and also potentially worrisome (for interop, I'm
>>> not saying Twisted is wrong here).
>>> 
>>> I think asyncio depends on the opposite: that if you add a callback to
>>> a Future that's ready it does *not* immediately run. Asyncio's promise
>>> is pretty strongly that callbacks are serialized (no callbacks running
>>> inside other callbacks). IIRC we experimented with other semantics and
>>> found that it was harder to reason about. (IMO if you *know* a Future
>>> is ready why add a callback to it rather than just calling the damn
>>> thing if that's what you want?)
>> 
>> I don't think Twisted is necessarily right here either.  You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred.  I like the property where you can do:
>> 
>> def test(test_case):
>>     a = asynchronously_something()
>>     test_case.this_must_not_have_a_result(a)
>>     cause_a_to_fire()
>>     test_case.assertEqual(test_case.this_must_have_a_result(a), something)
>> 
>> but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does.  So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without.  You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
>> 
>> Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?
> 
> Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it.  So the main use-case is testing.
> 
>> I suppose Deferred has a method to mark it done.
> 
> Yep; ".callback".
> 
>> Does that immediately run the callbacks?
> 
> It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.
> 
> There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.
> 
> So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
> 
>> Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
> 
> No.  A Deferred that has been called back stays called back; callbacking it again is always an error.  However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
> 
>> (I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
> 
> Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).
> 
> -glyph
> 
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/


From ben at bendarnell.com  Wed Jun  8 14:17:54 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Wed, 8 Jun 2016 14:17:54 -0400
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <36BA2A9F-9F14-4CE6-976F-17689C405C09@gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
 <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
 <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>
 <2F7C1C4A-39F4-4A58-AA25-96E61C20DCCB@twistedmatrix.com>
 <CAP7+vJJtzswz+LE1qgS5TJX=UxafB=VOiZ_GwXRzBY02-FaiOg@mail.gmail.com>
 <5BD7621E-5D57-4AB4-9A90-647124ED4F56@twistedmatrix.com>
 <36BA2A9F-9F14-4CE6-976F-17689C405C09@gmail.com>
Message-ID: <CAFkYKJ5PGXcBpK8oifN4haLQU-yhyTHkgQS5Upo0zZq2LG6oBA@mail.gmail.com>

On Wed, Jun 8, 2016 at 1:52 PM, Yury Selivanov <yselivanov at gmail.com> wrote:

> Glyph, Ben, Amber,
>
> So what?s the resolution on Future.__isfuture__ and fixing the
> isinstance(obj, Future) checks from asyncio?
>
> 3.5.2 RC is only few days away, I can still make the change if
> it?s a blocker for Twisted and Tornado.
>

None of this is blocking Tornado - we shipped asyncio integration six
months ago. There's some room for improvement, but I don't think there's a
clear enough mandate to squeeze something in for this release. I'd rather
take the time to sort out a more complete plan before the next release.

-Ben


>
> Yury
>
> > On Jun 6, 2016, at 5:35 PM, Glyph <glyph at twistedmatrix.com> wrote:
> >
> >>
> >> On Jun 6, 2016, at 14:21, Guido van Rossum <guido at python.org> wrote:
> >>
> >> On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com> wrote:
> >>
> >>
> >>> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org> wrote:
> >>>
> >>> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com>
> wrote:
> >>>>
> >>>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com> wrote:
> >>>>
> >>>> If things are so sensitive to minor changes in timing, doesn't that
> set the
> >>>> bar impossibly high for interoperability?
> >>>>
> >>>>
> >>>> The sensitivity is not to changes in timing - i.e. when the
> wall-clock runs,
> >>>> or ordering of non-deterministically ordered events - but rather to
> >>>> reentrancy - whether certain things complete synchronously while the
> caller
> >>>> is still on the stack and can depend on them having done so upon
> return.
> >>>>
> >>>> The recommended way of writing tests within Twisted these days depends
> >>>> heavily on `.callback? synchronously resolving a Deferred, which is
> what
> >>>> adding a call_soon breaks.
> >>>
> >>> That's interesting, and also potentially worrisome (for interop, I'm
> >>> not saying Twisted is wrong here).
> >>>
> >>> I think asyncio depends on the opposite: that if you add a callback to
> >>> a Future that's ready it does *not* immediately run. Asyncio's promise
> >>> is pretty strongly that callbacks are serialized (no callbacks running
> >>> inside other callbacks). IIRC we experimented with other semantics and
> >>> found that it was harder to reason about. (IMO if you *know* a Future
> >>> is ready why add a callback to it rather than just calling the damn
> >>> thing if that's what you want?)
> >>
> >> I don't think Twisted is necessarily right here either.  You're
> absolutely right that it's easier to reason about reentrancy in some cases
> if you have a might-be-fired-might-not Future vs. the same sort of
> Deferred.  I like the property where you can do:
> >>
> >> def test(test_case):
> >>     a = asynchronously_something()
> >>     test_case.this_must_not_have_a_result(a)
> >>     cause_a_to_fire()
> >>     test_case.assertEqual(test_case.this_must_have_a_result(a),
> something)
> >>
> >> but this is (somewhat) opposed to the fact that call_soon means you
> never get the nasty surprise where the callback added in the middle of a
> function gets run before the rest of it does.  So I think there are good
> properties in both cases and given some thought it is probably possible to
> map between them, but Deferred is lower-level here in the sense that it
> provides a way to do this both with the event loop and without.  You can
> always call_soon(deferred.callback) but you can't go the other way and
> force a Future to resolve synchronously - right?
> >>
> >> Right. I'm still unclear on what the compelling use case for that is
> (other than that Twisted has always done this). Is it performance? Is it
> callback ordering?
> >
> > Very, very early in the development of Deferreds, they worked the way
> Futures do; we changed it mainly to reduce coupling to the event loop so
> that we could test general-purpose algorithms (like gatherResults) without
> needing to spin an event loop to do it.  So the main use-case is testing.
> >
> >> I suppose Deferred has a method to mark it done.
> >
> > Yep; ".callback".
> >
> >> Does that immediately run the callbacks?
> >
> > It runs callbacks up to the point that the first one returns a Deferred,
> and then it waits for that one to be fired to continue running the chain.
> >
> > There is an exception here, where Deferred effectively opts in to
> Future-like behavior in a very specific case: if you are recursively giving
> results to a Deferred X that would un-block a Deferred Y inside a callback
> on Y, Y will not execute its own callbacks reentrantly; it waits until the
> current callback is done.
> >
> > So while the semantics of .callback() on a Deferred are clear-cut with
> respect to that Deferred itself, "continue any Deferreds waiting upon it"
> is a callback-like structure that is slightly squirrely in a very
> call_soon-like way to avoid surprise reentrancy and RecursionError
> explosions when having a structure like an asynchronous 'for' loop.
> >
> >> Or does it come in two flavors? Can a Deferred that's marked done ever
> revert back to being not done?
> >
> > No.  A Deferred that has been called back stays called back; callbacking
> it again is always an error.  However, it may pause running its chain if
> you return another Deferred in the middle someplace; the way to resume it
> is to give the inner Deferred a result; the outer one cannot be otherwise
> compelled to continue.
> >
> >> (I believe I once read the Deferred code enough to be able to find the
> answers, but I'm afraid I've never really needed what I learned then, so
> I've forgotten...)
> >
> > Happy to fill in these blanks; they're (mostly, modulo the weird
> exception for callbacks-in-callbacks) straightforward :).
> >
> > -glyph
> >
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160608/33480cb6/attachment-0001.html>

From glyph at twistedmatrix.com  Wed Jun  8 17:15:10 2016
From: glyph at twistedmatrix.com (Glyph)
Date: Wed, 8 Jun 2016 14:15:10 -0700
Subject: [Async-sig] asyncio <-> twisted interoperability
In-Reply-To: <CAFkYKJ5PGXcBpK8oifN4haLQU-yhyTHkgQS5Upo0zZq2LG6oBA@mail.gmail.com>
References: <897FD900-B789-4A3F-B9D6-2884FC516E6E@gmail.com>
 <9204F48E-D96A-48CB-AD82-2C6ADDF17BE3@langa.pl>
 <1B855C74-82B9-45DE-9FD8-D754326C7DCA@gmail.com>
 <CAFkYKJ6G6EdjCZxv4iJdb_UJfQM3F21hpyxFSWfTMiZSYUk_Sg@mail.gmail.com>
 <27A4544A-8DA4-41E1-93D5-440B2251ABA0@twistedmatrix.com>
 <CAP7+vJJzZx_nAfyrKwkpGWHGXKVT_d5EqfK+ah8nYDe8aiLuAg@mail.gmail.com>
 <2F7C1C4A-39F4-4A58-AA25-96E61C20DCCB@twistedmatrix.com>
 <CAP7+vJJtzswz+LE1qgS5TJX=UxafB=VOiZ_GwXRzBY02-FaiOg@mail.gmail.com>
 <5BD7621E-5D57-4AB4-9A90-647124ED4F56@twistedmatrix.com>
 <36BA2A9F-9F14-4CE6-976F-17689C405C09@gmail.com>
 <CAFkYKJ5PGXcBpK8oifN4haLQU-yhyTHkgQS5Upo0zZq2LG6oBA@mail.gmail.com>
Message-ID: <EEA3A79C-5E0B-4B28-88A7-9F6517B0771F@twistedmatrix.com>

Similarly, txtulip has existed for some time.  It would definitely be more performant and simpler if we could skip the subclass check, but we don't need this to happen in this cycle.

-glyph

> On Jun 8, 2016, at 11:17 AM, Ben Darnell <ben at bendarnell.com> wrote:
> 
> On Wed, Jun 8, 2016 at 1:52 PM, Yury Selivanov <yselivanov at gmail.com <mailto:yselivanov at gmail.com>> wrote:
> Glyph, Ben, Amber,
> 
> So what?s the resolution on Future.__isfuture__ and fixing the
> isinstance(obj, Future) checks from asyncio?
> 
> 3.5.2 RC is only few days away, I can still make the change if
> it?s a blocker for Twisted and Tornado.
> 
> None of this is blocking Tornado - we shipped asyncio integration six months ago. There's some room for improvement, but I don't think there's a clear enough mandate to squeeze something in for this release. I'd rather take the time to sort out a more complete plan before the next release. 
> 
> -Ben
>  
> 
> Yury
> 
> > On Jun 6, 2016, at 5:35 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> >
> >>
> >> On Jun 6, 2016, at 14:21, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote:
> >>
> >> On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> >>
> >>
> >>> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote:
> >>>
> >>> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> >>>>
> >>>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com <mailto:ben at bendarnell.com>> wrote:
> >>>>
> >>>> If things are so sensitive to minor changes in timing, doesn't that set the
> >>>> bar impossibly high for interoperability?
> >>>>
> >>>>
> >>>> The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
> >>>> or ordering of non-deterministically ordered events - but rather to
> >>>> reentrancy - whether certain things complete synchronously while the caller
> >>>> is still on the stack and can depend on them having done so upon return.
> >>>>
> >>>> The recommended way of writing tests within Twisted these days depends
> >>>> heavily on `.callback? synchronously resolving a Deferred, which is what
> >>>> adding a call_soon breaks.
> >>>
> >>> That's interesting, and also potentially worrisome (for interop, I'm
> >>> not saying Twisted is wrong here).
> >>>
> >>> I think asyncio depends on the opposite: that if you add a callback to
> >>> a Future that's ready it does *not* immediately run. Asyncio's promise
> >>> is pretty strongly that callbacks are serialized (no callbacks running
> >>> inside other callbacks). IIRC we experimented with other semantics and
> >>> found that it was harder to reason about. (IMO if you *know* a Future
> >>> is ready why add a callback to it rather than just calling the damn
> >>> thing if that's what you want?)
> >>
> >> I don't think Twisted is necessarily right here either.  You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred.  I like the property where you can do:
> >>
> >> def test(test_case):
> >>     a = asynchronously_something()
> >>     test_case.this_must_not_have_a_result(a)
> >>     cause_a_to_fire()
> >>     test_case.assertEqual(test_case.this_must_have_a_result(a), something)
> >>
> >> but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does.  So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without.  You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
> >>
> >> Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?
> >
> > Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it.  So the main use-case is testing.
> >
> >> I suppose Deferred has a method to mark it done.
> >
> > Yep; ".callback".
> >
> >> Does that immediately run the callbacks?
> >
> > It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.
> >
> > There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.
> >
> > So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
> >
> >> Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
> >
> > No.  A Deferred that has been called back stays called back; callbacking it again is always an error.  However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
> >
> >> (I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
> >
> > Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).
> >
> > -glyph
> >
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org <mailto:Async-sig at python.org>
> > https://mail.python.org/mailman/listinfo/async-sig <https://mail.python.org/mailman/listinfo/async-sig>
> > Code of Conduct: https://www.python.org/psf/codeofconduct/ <https://www.python.org/psf/codeofconduct/>
> 
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org <mailto:Async-sig at python.org>
> https://mail.python.org/mailman/listinfo/async-sig <https://mail.python.org/mailman/listinfo/async-sig>
> Code of Conduct: https://www.python.org/psf/codeofconduct/ <https://www.python.org/psf/codeofconduct/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160608/8cafeea4/attachment.html>

From andrew.svetlov at gmail.com  Thu Jun  9 00:38:20 2016
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Thu, 09 Jun 2016 04:38:20 +0000
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <37D25E18-A67E-4639-ABE8-9A9BF468753E@gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <EB1F6EE7-4A2F-4C41-AA79-E31338A7166B@gmail.com>
 <CAPJVwBnj4E2dmRQ+EGWGBCTSR+xeJ7ogAPfob2_N61jfEntVKg@mail.gmail.com>
 <CAPJVwB=_2OanSA4=BWBihGbFWdpdOq3q+pc0atEOWjxNjF9L_Q@mail.gmail.com>
 <CAPJVwB=K64OJposYkx5SqvGuB3DERmU0upvao-th_c6TU9Ys_A@mail.gmail.com>
 <CAL3CFcWwC96WWa2B46kOR8U_UZSj7Vyi6Aw7FxPtHZHC_GT7WA@mail.gmail.com>
 <37D25E18-A67E-4639-ABE8-9A9BF468753E@gmail.com>
Message-ID: <CAL3CFcV6Nxn-57c28sLB-KYhr=PygpZCWnH7XzxXUcV1mzTL6A@mail.gmail.com>

My first thought was to use `async with` but I found that just `with` also
works.

Sorry, but I don't follow how async context manager may be better.

On Wed, Jun 8, 2016 at 10:49 AM Yury Selivanov <yselivanov at gmail.com> wrote:

>
> > On Jun 8, 2016, at 1:15 PM, Andrew Svetlov <andrew.svetlov at gmail.com>
> wrote:
> >
> > asyncio.timeout behaves like asyncio.wait_for BTW.
> > It cancels inner tasks and sends asyncio.TimeoutError to outer.
>
> Right.  And I think it?s a totally normal behaviour for asyncio.
>
> The only thing I?d fix in asyncio.timeout() (aside from resolving
> Ben?s questions) is to make it use ?async with? statement.
>
> Yury
>
> --
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160609/3dd7d8ce/attachment-0001.html>

From yselivanov at gmail.com  Thu Jun  9 00:40:24 2016
From: yselivanov at gmail.com (Yury Selivanov)
Date: Thu, 9 Jun 2016 00:40:24 -0400
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAL3CFcV6Nxn-57c28sLB-KYhr=PygpZCWnH7XzxXUcV1mzTL6A@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <EB1F6EE7-4A2F-4C41-AA79-E31338A7166B@gmail.com>
 <CAPJVwBnj4E2dmRQ+EGWGBCTSR+xeJ7ogAPfob2_N61jfEntVKg@mail.gmail.com>
 <CAPJVwB=_2OanSA4=BWBihGbFWdpdOq3q+pc0atEOWjxNjF9L_Q@mail.gmail.com>
 <CAPJVwB=K64OJposYkx5SqvGuB3DERmU0upvao-th_c6TU9Ys_A@mail.gmail.com>
 <CAL3CFcWwC96WWa2B46kOR8U_UZSj7Vyi6Aw7FxPtHZHC_GT7WA@mail.gmail.com>
 <37D25E18-A67E-4639-ABE8-9A9BF468753E@gmail.com>
 <CAL3CFcV6Nxn-57c28sLB-KYhr=PygpZCWnH7XzxXUcV1mzTL6A@mail.gmail.com>
Message-ID: <1CA5F7F9-7FD2-4006-8DED-419AC6476F7D@gmail.com>


> On Jun 9, 2016, at 12:38 AM, Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
> 
> My first thought was to use `async with` but I found that just `with` also works.
> 
> Sorry, but I don't follow how async context manager may be better.

It would make asyncio.timeout() usable only inside coroutines.  Using
it outside of a coroutine is an error anyways.

Yury


From ben at bendarnell.com  Sun Jun 12 12:00:58 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Sun, 12 Jun 2016 09:00:58 -0700
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAPJVwBmL4WC1-0ip7zdK0o3tbK2FkwzWxWd=N-Y9TpXU3Pe=yg@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <CAP7+vJLnx4yYKBmmQGknVBMTLPgMCngzXNuihj9LXMiHUzRboQ@mail.gmail.com>
 <CAFkYKJ5ci9mgWPCsv=H1yHOF9H386ewh0kybQF8nYjMT1MzW9g@mail.gmail.com>
 <CAPJVwBmL4WC1-0ip7zdK0o3tbK2FkwzWxWd=N-Y9TpXU3Pe=yg@mail.gmail.com>
Message-ID: <CAFkYKJ57cMfo_W78m6r7p1OWWO0yNg-VUqdsoFVSYxLNvX4f5w@mail.gmail.com>

On Sun, Jun 5, 2016 at 11:38 AM, Nathaniel Smith <njs at pobox.com> wrote:

>
> And AFAICT the tornado.gen.with_timeout wrapper doesn't deal with
> coroutines/coroutine runners at all?


It doesn't currently, but this is a bug. It should be creating a coroutine
runner for the input task if it is not already a Future.


> I just read the source, but I
> couldn't fully follow it. If I write
>   async def do_stuff():
>       try:
>           await aiohttp.get("https://python.org")
>       except:
>           print("oops, cleaning up")
>           raise
>       else:
>           print("did it!")
>   await tornado.gen.with_timeout(timedelta(seconds=0.0001), do_stuff())
> and the timeout fires, then is the output "oops, cleaning up", "did
> it!", or nothing at all?
>

This would not produce any output: Tornado doesn't do asyncio-style
cancellation (I've had enough bad experiences with KeyboardInterrupt
cropping up at unexpected times). The inner task has its own coroutine
runner and will continue after the outer task has stopped waiting for it.
This allows the Future from the inner task to be reused (either in the same
outer task or in another task), but it does require that some lower-level
timeout mechanism also be present to prevent timed-out tasks from
accumulating forever. This is not ideal and it would be good to allow the
inner task to find out its been canceled (the reusability of Futures only
matters in rare cases).

-Ben


>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160612/80dfbc84/attachment.html>

From ben at bendarnell.com  Sun Jun 12 12:15:38 2016
From: ben at bendarnell.com (Ben Darnell)
Date: Sun, 12 Jun 2016 09:15:38 -0700
Subject: [Async-sig] asyncio.timeout() is not portable
In-Reply-To: <CAL3CFcV6Nxn-57c28sLB-KYhr=PygpZCWnH7XzxXUcV1mzTL6A@mail.gmail.com>
References: <CAFkYKJ6j2YyN0HHYzVtJbziDH8ejw6p13-KCF7EM8oibQw_2Ww@mail.gmail.com>
 <EB1F6EE7-4A2F-4C41-AA79-E31338A7166B@gmail.com>
 <CAPJVwBnj4E2dmRQ+EGWGBCTSR+xeJ7ogAPfob2_N61jfEntVKg@mail.gmail.com>
 <CAPJVwB=_2OanSA4=BWBihGbFWdpdOq3q+pc0atEOWjxNjF9L_Q@mail.gmail.com>
 <CAPJVwB=K64OJposYkx5SqvGuB3DERmU0upvao-th_c6TU9Ys_A@mail.gmail.com>
 <CAL3CFcWwC96WWa2B46kOR8U_UZSj7Vyi6Aw7FxPtHZHC_GT7WA@mail.gmail.com>
 <37D25E18-A67E-4639-ABE8-9A9BF468753E@gmail.com>
 <CAL3CFcV6Nxn-57c28sLB-KYhr=PygpZCWnH7XzxXUcV1mzTL6A@mail.gmail.com>
Message-ID: <CAFkYKJ7S1SVhby6CesugvTjNr=x5CgwgJfpz1Wrrtdr4uB3mkw@mail.gmail.com>

On Wed, Jun 8, 2016 at 9:38 PM, Andrew Svetlov <andrew.svetlov at gmail.com>
wrote:

> My first thought was to use `async with` but I found that just `with` also
> works.
>

> Sorry, but I don't follow how async context manager may be better.
>

`async with` communicates directly with the coroutine runner. Plain `with`
requires that the coroutine runner be discoverable in some other way (via a
global). Either way, for `asyncio.timeout` to be portable to other
coroutine runners, we must define a protocol that coroutine runners may
implement.

If we use `async with`, that protocol is something like "if the coroutine
yields an instance of asyncio.TimeoutSetter, the runner should set its
timeout accordingly".

If we use plain `with`, we would need to move asyncio.Task._current_task
somewhere public (an attribute of EventLoop? a global threading.local?) and
the protocol would be that each coroutine runner must register itself there
while it is on the top of the stack and implement a `set_timeout()` method.

I believe the `async with` solution will perform better (an isinstance()
check on every yielded object, plus one extra yielded object when the
timeout feature is used, versus at least two operations on a central
registry every time the coroutine runner enters or leaves the stack), and
it also degrades more gracefully (the TimeoutSetter object could be a
subclass of Future that resolves instantly so runners that don't understand
it can simply ignore it. A runner that doesn't participate in the
current_task registry may leave the wrong task at the top of the stack).

-Ben


>
> On Wed, Jun 8, 2016 at 10:49 AM Yury Selivanov <yselivanov at gmail.com>
> wrote:
>
>>
>> > On Jun 8, 2016, at 1:15 PM, Andrew Svetlov <andrew.svetlov at gmail.com>
>> wrote:
>> >
>> > asyncio.timeout behaves like asyncio.wait_for BTW.
>> > It cancels inner tasks and sends asyncio.TimeoutError to outer.
>>
>> Right.  And I think it?s a totally normal behaviour for asyncio.
>>
>> The only thing I?d fix in asyncio.timeout() (aside from resolving
>> Ben?s questions) is to make it use ?async with? statement.
>>
>> Yury
>>
>> --
> Thanks,
> Andrew Svetlov
>
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160612/1648d849/attachment.html>

From lukasz at langa.pl  Mon Jun 13 03:39:45 2016
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Mon, 13 Jun 2016 00:39:45 -0700
Subject: [Async-sig] async executors
In-Reply-To: <87wpm2332c.fsf@grothesque.org>
References: <87wpm2332c.fsf@grothesque.org>
Message-ID: <D9701C10-50D0-46A1-A888-B0F8AEE76950@langa.pl>


> On Jun 6, 2016, at 5:38 AM, Christoph Groth <christoph at grothesque.org> wrote:
> 
> Why could such an async executor be interesting?

Yes, definitely interesting, please release it as a package on PyPI. It?s hard to say whether it?s going to be integrated with a future version or asyncio but releasing it independently and announcing it is the easiest way to learn.

I looked at your proposed changes on python-ideas and they look good. Having this up on GitHub with unit tests would be a great start of the conversation. We might be able to gather some real world data from using the builtin PPE and the pure asyncio one.

--
Lukasz Langa | Facebook
Production Engineer | The Ministry of Silly Walks
(+1) 650-681-7811

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160613/cffa13a0/attachment.sig>

From christoph at grothesque.org  Mon Jun 13 12:10:30 2016
From: christoph at grothesque.org (Christoph Groth)
Date: Mon, 13 Jun 2016 18:10:30 +0200
Subject: [Async-sig] async executors
References: <87wpm2332c.fsf@grothesque.org>
 <DB5F2662-C11E-4700-900C-4991FB400387@gmail.com>
Message-ID: <87fushrrwp.fsf@grothesque.org>

Hi Yury,

Thanks for your insightful reply.

>> ? Performance is better, especially when many worker processes 
>> are  involved, because of the simpler code that uses less 
>> locking  (the only locking that remains is inside the 
>> multiprocessing  module).
>
> Yes, it will be better, but the performance improvements will be 
> visible only for 100s or 1000s of processes.  And only if they 
> are executing short-living tasks.
>
> Is there a real need for a bit faster process pool?

You are right of course, 1000s of processes is not a realistic 
application of concurrent.futures.ProcessPoolExecutor on a normal 
computer.

When I started playing with async executor I had computing 
clusters in mind.  There exist several concurrent.futures-like 
libraries that work with computing clusters and support 1000s of 
worker processes (e.g. ipyparallel, distribute, SCOOP).  Several 
of these packages use async programming (e.g. tornado or greenlet) 
for their internals (schedulers, controlers, etc.), but the 
futures that they provide use locking, just like that of 
concurrent.futures.

In order to estimate whether an async executor could be useful for 
cluster-like workloads, I did some tests on my laptop with many 
worker processes that do mostly nothing.  The result is that using 
asyncio's run_in_executor allows to process up to 1000 tasks per 
second.  Using aexecutor, this number grows to 5000.  These two 
numbers seem reasonably robust when, for example, the number of 
workers is varied.

That factor 5 difference is not enormous, but perhaps there's some 
potential for improvement (optimization, curio?).  Let's try a 
real-world check: I am often using a cluster of 1000 cores, and 
each core is about 50% as fast as the core in my laptop.  So 
run_in_executor() will be overcharged if the average task takes 
less than 2 seconds to run.  This doesn't sound like a terrible 
restriction, one would certainly try to have tasks that run longer 
than that.  But I can certainly imagine useful workloads where 
tasks run less than 2 seconds and the parameters and results of 
each task are only a few numbers at most, so that the 
communication bandwidth shouldn't be a limit.

>> Based on concurrent.futures.ProcessPoolExecutor, I?ve made a 
>> proof-of-concept implementation [3] that demonstrates that the 
>> idea works.  (There have been some improvements compared to the 
>> version that I posted on python-ideas.)
>
> I think you should stabilize the code base, and release it as a 
> module on PyPI.

I will do that, once I have some confidence that there are no 
obvious blunders in it (see further below).

>> I would be grateful if any asyncio experts could have a look at 
>> the part of the main loop of the process management coroutine 
>> where the coroutine awaits new data [4].  Currently, I am using 
>> an asyncio.Event that is set by a callback via asyncio?s 
>> add_reader().  Is this the most natural way to do it currently? 
>
> I think you can use asyncio.Future for resuming that coroutine.

I'm not sure what you mean.  In the current code (line in 
concurrent.futures), a multiprocessing.SimpleQueue instance is 
used to receive results from the workers.  That queue has an 
attribute _reader._handle that is just a file descriptor.

I use BaseEventLoop.add_reader() to add a callback for that file 
descriptor.  The callback sets an asyncio.Event and the main loop 
of _queue_management_worker() waits for this event.

How could I use asyncio.Future here?

One problem with my current solution is that the event gets set 
also when there is no data in the queue yet.  That's why I use the 
method poll() of the reader to verify if there's anything to be 
read and if there isn't, the event is cleared again and the 
waiting resumes.  The spurious events could be an indication that 
there is a better way of solving the problem, hence my original 
question.

Thanks,
Christoph
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160613/5c1fabaf/attachment.sig>

From christoph at grothesque.org  Mon Jun 13 12:22:38 2016
From: christoph at grothesque.org (Christoph Groth)
Date: Mon, 13 Jun 2016 18:22:38 +0200
Subject: [Async-sig] async executors
In-Reply-To: <041210F2-1DC3-4543-83E8-F51DE8E807A4@langa.pl>
 (=?utf-8?Q?=22=C5=81ukasz?= Langa"'s
 message of "Mon, 6 Jun 2016 11:32:58 -0700")
References: <87wpm2332c.fsf@grothesque.org>
 <041210F2-1DC3-4543-83E8-F51DE8E807A4@langa.pl>
Message-ID: <878ty9rrch.fsf@grothesque.org>

?ukasz Langa wrote:

>> As far as I can see the original 
>> concurrent.futures.ProcessPoolExecutor has no advantage when 
>> used in asyncio-based programs except when some coroutine 
>> blocks for long enough for the call queue to become empty. 
>> (But hey, async programming is about non-blocking coroutines!)
>
> That?s incorrect. The ProcessPoolExecutor is a way for you to 
> run CPU-intensive tasks without affecting the responsiveness of 
> the main event loop in your program. As long as you keep your 
> arguments small (pickling/unpickling big data structures is 
> pretty detrimental to performance here) and the duration of the 
> task is non-trivial, the child processes enable you to use more 
> CPU cores, and since they have their own GILs, this doesn't 
> affect your loop at all, unlike using a thread pool.

I was comparing the (possible) advantages of the traditional 
concurrent.futures.ProcessPoolExecutor with the new 
aexecutor.ProcessPoolExecutor.  Both use processes for the 
workers, so the GIL is not a problem for either.

Christoph
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160613/7cbf8c11/attachment.sig>