[Async-sig] async/sync library reusage

Mon Jun 12 14:50:26 EDT 2017

Honestly I think we're in agreement. There's never a use for one loop
running while another is the default. There are some rare use cases for
multiple loops running but before the mentioned commit it was up to the app
to ensure to switch the default loop when running a loop. The commit took
the ability to screw up there out of the user's hand.

On Mon, Jun 12, 2017 at 9:57 AM, Andrew Svetlov <andrew.svetlov at gmail.com>
wrote:

> Yes, but with one exception: default event loop created on module import
> stage might co-exist with a loop created for test.
> It leads to mystic hangs, you know.
> Please recall code like:
>     class A:
>          mongodb = motor.motor_asyncio.AsyncIOMotorClient()
>
> On Mon, Jun 12, 2017 at 7:37 PM Guido van Rossum <guido at python.org> wrote:
>
>> Yes, but not co-existing, I hope!
>>
>> On Mon, Jun 12, 2017 at 9:25 AM, Andrew Svetlov <andrew.svetlov at gmail.com
>> > wrote:
>>
>>> Unit tests at least. Running every test in own loop is crucial fro tests
>>> isolation.
>>>
>>> On Mon, Jun 12, 2017 at 7:04 PM Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> Multiple loops in the same thread is purely theoretical -- the API
>>>> allows it but there's no use case. It might be necessary if a platform has
>>>> a UI-only event loop that cannot be extended to do I/O -- the only solution
>>>> to do background I/O might be to alternate between two loops. (Though in
>>>> that case I would still prefer a thread for the background I/O.)
>>>>
>>>> On Mon, Jun 12, 2017 at 8:49 AM, Pau Freixes <pfreixes at gmail.com>
>>>> wrote:
>>>>
>>>>> And what about the rationale of having multiple loop instances in the
>>>>> same thread switching btw them. Im still trying to find out what patterns
>>>>> need this... Do you have an example?
>>>>>
>>>>> Btw thanks for the first explanation
>>>>>
>>>>> El 12/06/2017 17:36, "Guido van Rossum" <guido at python.org> escribió:
>>>>>
>>>>>> In theory it's possible to create two event loops (using
>>>>>> new_event_loop()), then set one as the default event loop (using
>>>>>> set_event_loop()), then run the other one (using run_forever() or
>>>>>> run_until_complete()). To tasks running in the latter event loop,
>>>>>> get_event_loop() would nevertheless return the former.
>>>>>>
>>>>>> On Mon, Jun 12, 2017 at 4:39 AM, Pau Freixes <pfreixes at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry a bit of topic, but I would like to figure out why older python
>>>>>>> versions, prior this commit [1], the get_event_loop is not considered
>>>>>>> deterministic
>>>>>>>
>>>>>>> does anybody know the reason behind this change?
>>>>>>>
>>>>>>>
>>>>>>> [1] https://github.com/python/cpython/commit/
>>>>>>> 600a349781bfa0a8239e1cb95fac29c7c4a3302e
>>>>>>>
>>>>>>> On Fri, Jun 9, 2017 at 6:07 PM, Ben Darnell <ben at bendarnell.com>
>>>>>>> wrote:
>>>>>>> > On Fri, Jun 9, 2017 at 11:51 AM Cory Benfield <cory at lukasa.co.uk>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> My concern with multiple loops boils down to the fact that urllib3
>>>>>>> >> supports being used in a multithreaded context where each thread
>>>>>>> can
>>>>>>> >> independently make forward progress on one request. To establish
>>>>>>> that with a
>>>>>>> >> synchronous codebase you either need one event loop per thread or
>>>>>>> you need
>>>>>>> >> to spawn a background thread on startup that owns the only event
>>>>>>> loop in the
>>>>>>> >> process.
>>>>>>> >
>>>>>>> >
>>>>>>> > Yeah, one event loop per thread is probably the way to go for
>>>>>>> integration
>>>>>>> > with synchronous codebases. A dedicated event loop thread may
>>>>>>> perform better
>>>>>>> > but libraries that spawn threads are problematic.
>>>>>>> >
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Generally speaking I’ve not had positive results with libraries
>>>>>>> spawning
>>>>>>> >> their own threads in Python. In my experience this has tended to
>>>>>>> lead to
>>>>>>> >> programs that deadlock mysteriously or that fail to terminate in
>>>>>>> the face of
>>>>>>> >> a Ctrl+C. So I tend to prefer to have users spawn their own
>>>>>>> threads, which
>>>>>>> >> would make me want a “one-event-loop-per-thread” model: hence,
>>>>>>> needing a
>>>>>>> >> loop parameter to pass around prior to 3.6.
>>>>>>> >
>>>>>>> >
>>>>>>> > You can avoid the loop parameter on older versions of asyncio (at
>>>>>>> least as
>>>>>>> > long as the default event loop policy is used) by manually setting
>>>>>>> your
>>>>>>> > event loop as current before calling run_until_complete (and
>>>>>>> resetting it
>>>>>>> > afterwards).
>>>>>>> >
>>>>>>> > Tornado's run_sync() method is equivalent to asyncio's
>>>>>>> run_until_complete(),
>>>>>>> > and Tornado supports multiple IOLoops in this way. We use this to
>>>>>>> expose a
>>>>>>> > synchronous version of our AsyncHTTPClient:
>>>>>>> > https://github.com/tornadoweb/tornado/blob/
>>>>>>> 62e47215ce12aee83f951758c96775a43e80475b/tornado/httpclient.py#L54
>>>>>>> >
>>>>>>> > -Ben
>>>>>>> >
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> I admit that my concerns here regarding libraries spawning their
>>>>>>> own
>>>>>>> >> threads may be overblown: after my series of negative experiences
>>>>>>> I
>>>>>>> >> basically never went back to that model, and it may be that the
>>>>>>> problems
>>>>>>> >> were more user-error than anything else. However, I feel
>>>>>>> comfortable saying
>>>>>>> >> that libraries spawning their own Python threads is definitely
>>>>>>> subtle and
>>>>>>> >> hard to get right, at the very least.
>>>>>>> >>
>>>>>>> >> Cory
>>>>>>> >> _______________________________________________
>>>>>>> >> Async-sig mailing list
>>>>>>> >> Async-sig at python.org
>>>>>>> >> https://mail.python.org/mailman/listinfo/async-sig
>>>>>>> >> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>>>> >
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > Async-sig mailing list
>>>>>>> > Async-sig at python.org
>>>>>>> > https://mail.python.org/mailman/listinfo/async-sig
>>>>>>> > Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> --pau
>>>>>>> _______________________________________________
>>>>>>> Async-sig mailing list
>>>>>>> Async-sig at python.org
>>>>>>> https://mail.python.org/mailman/listinfo/async-sig
>>>>>>> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --Guido van Rossum (python.org/~guido)
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --Guido van Rossum (python.org/~guido)
>>>> _______________________________________________
>>>> Async-sig mailing list
>>>> Async-sig at python.org
>>>> https://mail.python.org/mailman/listinfo/async-sig
>>>> Code of Conduct: https://www.python.org/psf/codeofconduct/
>>>>
>>> --
>>> Thanks,
>>> Andrew Svetlov
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
> --
> Thanks,
> Andrew Svetlov
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20170612/30425c8a/attachment-0001.html>