[issue31814] subprocess_fork_exec more stable with vfork

[Albert Zeyer]

New submission from Albert Zeyer <albzey at googlemail.com>:

subprocess_fork_exec currently calls fork().

I propose to use vfork() or posix_spawn() or syscall(SYS_clone, SIGCHLD, 0) instead if possible and if there is no preexec_fn. The difference would be that fork() will call any atfork handlers (registered via pthread_atfork()), while the suggested calls would not.

There are cases where atfork handlers are registered which are not save to be called e.g. in multi-threaded environments. In the case of subprocess_fork_exec without preexec_fn, there is no need to call those atfork handlers, so avoiding this could avoid potential problems. It's maybe acceptable if a pure fork() without exec() doesn't work in this case anymore, but there is no reason that a fork()+exec() should not work in any such cases. This is fixed by my proposed solution.

An example case is OpenBlas and OpenMP, which registers an atfork handler which is safe to be called if there are other threads running.
See here:
https://github.com/tensorflow/tensorflow/issues/13802
https://github.com/xianyi/OpenBLAS/issues/240
https://trac.sagemath.org/ticket/22021

About fork+exec without the atfork handlers, see here for alternatives (like vfork):
https://stackoverflow.com/questions/46810597/forkexec-without-atfork-handlers/

----------
components: Interpreter Core
messages: 304587
nosy: Albert.Zeyer
priority: normal
severity: normal
status: open
title: subprocess_fork_exec more stable with vfork
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue31814>
_______________________________________