[Async-sig] async/sync library reusage

[Guido van Rossum]

On Fri, Jun 9, 2017 at 8:51 AM, Cory Benfield <cory at lukasa.co.uk> wrote:

>
> On 9 Jun 2017, at 16:40, Guido van Rossum <guido at python.org> wrote:
>
> Also, I don't think the situation with explicitly passing loop= is so
> terrible as you seem to think. If you rely on the default event loop, you
> rely on there *being* a default event loop, but there will always be one
> unless an app goes out of its way to create an event loop and then make it
> not the default loop. Only the asyncio tests do that. There are a few
> things you can't do unless you pass an event loop (such as scheduling
> callbacks before the event loop is started) but other than that it's really
> not such a big deal as people seem to think it is. (You mostly see the
> pattern because asyncio itself uses that pattern, because it needs to be
> robust for the extreme use case where someone *does* hide the active event
> loop. But there will never be two active event loops.)
>
>
> My concern with multiple loops boils down to the fact that urllib3
> supports being used in a multithreaded context where each thread can
> independently make forward progress on one request. To establish that with
> a synchronous codebase you either need one event loop per thread or you
> need to spawn a background thread on startup that owns the only event loop
> in the process.
>
> Generally speaking I’ve not had positive results with libraries spawning
> their own threads in Python. In my experience this has tended to lead to
> programs that deadlock mysteriously or that fail to terminate in the face
> of a Ctrl+C. So I tend to prefer to have users spawn their own threads,
> which would make me want a “one-event-loop-per-thread” model: hence,
> needing a loop parameter to pass around prior to 3.6.
>
> I admit that my concerns here regarding libraries spawning their own
> threads may be overblown: after my series of negative experiences I
> basically never went back to that model, and it may be that the problems
> were more user-error than anything else. However, I feel comfortable saying
> that libraries spawning their own Python threads is definitely subtle and
> hard to get right, at the very least.


At least one of us is still confused. The one-event-loop-per-thread model
is supported in asyncio without passing the loop around explicitly. The
get_event_loop() implementation stores all its state in thread-locals
instance, so it returns the thread's event loop. (Because this is an
"advanced" model, you have to explicitly create the event loop with
new_event_loop() and make it the default loop for the thread with
set_event_loop().)

All in all, I'm a bit curious why you would need to use asyncio at all when
you've got a thread per request anyway.

I agree there are problems with threads that are hidden from an app. Hence
asyncio allows you to set the executor where it runs things you pass to
run_in_executor() (including some of its own, esp. getaddrinfo()).

One note about the one-event-loop-per-thread model: threads should be very
cautious touching each other's event loops. This should only be done
usingcall_soon_threadsafe()!

--Guido

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20170609/e4ce21f8/attachment.html>