[Async-sig] avoiding accidentally calling blocking code

[Andrew Svetlov]

loop.slow_callback_duration + PYTHONASYNCIODEBUG=1
Does it fork for you?
Do you need extra info?

On Tue, Feb 13, 2018 at 7:07 AM Dima Tisnek <dimaqq at gmail.com> wrote:

> Let me try to answer the question behind the question.
>
> Like any code validation, it's a healthy mix of:
> * unit tests [perhaps with
> mock.patch_all_known_blocking_calls(side_effect=Exception)]
> * good judgement [open("foo").read() technically blocks, but only a
> problem on network filesystems]
> * monitoring prod [at least to ensure that tests correspond to prod]
> * peer review
>
>
> Specific issues you've raised:
>
> re: detecting blocking calls ahead of time.
> That's a hard problem (insert quip about Turing completeness)
> Perhaps it can be alleviated by function annotations, but that's
> dangerously close to Java way...
>
> re: detecting blocking calls at run time.
> Let's say there's a per-thread global "blocking lock" that's managed
> by event loop executor and it's in "allowed" state when executor has
> no runnables and wishes to select/poll/epoll and it's in "denied"
> state when user code is running.
> Some calls can be caught (e.g. check fd fcntl bits before any
> recv/send/...)
> Some calls can never be caught (anything via ctypes or extensions in
> general)
> Case in point: what do you propose to do about socket.gethostbyname()?
>
> re: after the fact.
> hack1. run a separate thread/process that inspects event loop thread's
> `wchan` value (Linux)
> hack2. same via taskstats
> hack3. detect event loop lag (false positives are cpu-bound tasks and
> overloaded system, but I think you'd like to detect those too)
>
> On 13 February 2018 at 12:10, Chris Jerdonek <chris.jerdonek at gmail.com>
> wrote:
> > Hi,
> >
> > This is a general Python async question I've had that I haven't seen a
> > discussion of. Do people know of any techniques other than manually
> > inspecting code line by line to find out if you're accidentally making
> > a raw call to a blocking function in a coroutine?
> >
> > Related to this, again, outside of inspecting the external source
> > code, is there any way to know if a function you're calling (e.g. in
> > someone else's library or in the standard lib) has the potential of
> > blocking?
> >
> > What would be the way of detecting something like this if it made its
> > way into "production"? What would be some of the symptoms?
> >
> > --Chris
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-- 
Thanks,
Andrew Svetlov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20180213/ba817ac9/attachment.html>