multiprocessing module backport from 3 to 2.7 - spawn feature

[Sturla Molden]

On 30/01/15 23:25, Marko Rauhamaa wrote:
> Sturla Molden <sturla.molden at gmail.com>:
>
>> Only a handful of POSIX functions are required to be "fork safe", i.e.
>> callable on each side of a fork without an exec.
>
> That is a pretty surprising statement. Forking without an exec is a
> routine way to do multiprocessing.
>
> I understand there are things to consider, but all system calls are
> available and safe.

POSIX says this:


- No asynchronous input or asynchronous output operations shall be 
inherited by the child process.

- A process shall be created with a single thread. If a multi-threaded 
process calls fork(), the new process shall contain a replica of the 
calling thread and its entire address space, possibly including the 
states of mutexes and other resources. Consequently, to avoid errors, 
the child process may only execute async-signal-safe operations until 
such time as one of the exec functions is called.

- Fork handlers may be established by means of the pthread_atfork() 
function in order to maintain application invariants across fork() calls.

- When the application calls fork() from a signal handler and any of the 
fork handlers registered by pthread_atfork() calls a function that is 
not asynch-signal-safe, the behavior is undefined.


Hence you must be very careful which functions you use after calling 
forking before you have called exec. Generally never use an API above 
POSIX, e.g. BLAS or Apple's CoreFoundation.



Apple said this when the problem with multiprocessing and Accelerate 
Framework first was discovered:


---------- Forwarded message ----------
From:  <devbugs at apple.com>
Date: 2012/8/2
Subject: Bug ID 11036478: Segfault when calling dgemm with Accelerate
/ GCD after in a forked process
To: ******@******


Hi Olivier,

Thank you for contacting us regarding Bug ID# 11036478.

Thank you for filing this bug report.

This usage of fork() is not supported on our platform.

For API outside of POSIX, including GCD and technologies like
Accelerate, we do not support usage on both sides of a fork(). For
this reason among others, use of fork() without exec is discouraged in
general in processes that use layers above POSIX.

We recommend that you either restrict usage of blas to the parent or
the child process but not both, or that you switch to using GCD or
pthreads rather than forking to create parallelism.



Also see this:

http://bugs.python.org/issue8713

https://mail.python.org/pipermail/python-ideas/2012-November/017930.html





Sturla