[Async-sig] avoiding accidentally calling blocking code

[Dima Tisnek]

Let me try to answer the question behind the question.

Like any code validation, it's a healthy mix of:
* unit tests [perhaps with
mock.patch_all_known_blocking_calls(side_effect=Exception)]
* good judgement [open("foo").read() technically blocks, but only a
problem on network filesystems]
* monitoring prod [at least to ensure that tests correspond to prod]
* peer review


Specific issues you've raised:

re: detecting blocking calls ahead of time.
That's a hard problem (insert quip about Turing completeness)
Perhaps it can be alleviated by function annotations, but that's
dangerously close to Java way...

re: detecting blocking calls at run time.
Let's say there's a per-thread global "blocking lock" that's managed
by event loop executor and it's in "allowed" state when executor has
no runnables and wishes to select/poll/epoll and it's in "denied"
state when user code is running.
Some calls can be caught (e.g. check fd fcntl bits before any recv/send/...)
Some calls can never be caught (anything via ctypes or extensions in general)
Case in point: what do you propose to do about socket.gethostbyname()?

re: after the fact.
hack1. run a separate thread/process that inspects event loop thread's
`wchan` value (Linux)
hack2. same via taskstats
hack3. detect event loop lag (false positives are cpu-bound tasks and
overloaded system, but I think you'd like to detect those too)

On 13 February 2018 at 12:10, Chris Jerdonek <chris.jerdonek at gmail.com> wrote:
> Hi,
>
> This is a general Python async question I've had that I haven't seen a
> discussion of. Do people know of any techniques other than manually
> inspecting code line by line to find out if you're accidentally making
> a raw call to a blocking function in a coroutine?
>
> Related to this, again, outside of inspecting the external source
> code, is there any way to know if a function you're calling (e.g. in
> someone else's library or in the standard lib) has the potential of
> blocking?
>
> What would be the way of detecting something like this if it made its
> way into "production"? What would be some of the symptoms?
>
> --Chris
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/