[issue21595] Creating many subprocess generates lots of internal BlockingIOError

[STINNER Victor]

STINNER Victor added the comment:

"Exception ignored when trying to write to the signal wakeup fd" message comes from the signal handler in Modules/signalmodule.c. The problem is that Python gets a lot of SIGCHLD signals (the test scripts creates +300 processes per second on my computer). The producer (signal handler writing the signal number into the "self" pipe) is faster than the consumer (BaseSelectorEventLoop._read_from_self callback).

Attached patch should reduce the risk of seeing the message "Exception ignored when trying to write to the signal wakeup fd". The patch reads all pending of the self pipe, instead of just trying to read a signal byte.

The test script doesn't write the error message anymore when the patch is applied (the script creates more than 300 processes per second).

The patch doesn't solve completly the issue. Other possible enhancements:

* Add a flag in the signal handler to notify that a signal was received, and write a single byte until the flag is reset to False. It would avoid to fill the pipe. It requires to implement a custom signal handler implemented in C, different from signal handlers of the Python module.

* Add an higher priority to callbacks of signal handlers. Asyncio doesn't support priority on callbacks right now.

* Increaze the size of the pipe. On Linux, it looks like "fcntl(fd, F_SETPIPE_SZ, size);" can be used. The maximum size is /proc/sys/fs/pipe-max-size (ex: 1 MB of my Fedora 20).

----------
keywords: +patch
nosy: +giampaolo.rodola, gvanrossum, haypo, pitrou, yselivanov
Added file: http://bugs.python.org/file35388/asyncio_read_from_self.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21595>
_______________________________________