[Python-ideas] solving multi-core Python

[Devin Jeanpierre]

On Wed, Jun 24, 2015 at 12:31 PM, Gregory P. Smith <greg at krypto.org> wrote:
> You cannot assume that fork() is safe on any OS as a general solution for
> anything.  This isn't a Windows specific problem, It simply cannot be relied
> upon in a general purpose library at all.  It is incompatible with threads.
>
> The ways fork() can be used safely are in top level application decisions:
> There must be a guarantee of no threads running before all forking is done.
> (thus the impossibility of relying on it as a mechanism to do anything
> useful in a generic library - you are a library, you don't know what the
> whole application is doing or when you were called as part of it)
>
> A concurrency model that assumes that it is fine to fork() and let child
> processes continue to execute is not usable by everyone. (ie:
> multiprocessing until http://bugs.python.org/issue8713 was implemented).

Another way of looking at it is that a concurrency model that assumes
it is fine to thread and let child threads continue to execute is not
usable by everyone.

IMO the lesson here is don't start threads *or* fork processes behind
the scenes without explicitly allowing your callers to override you,
so that the top level app can orchestrate everything appropriately.
This is especially important in Python, where forking is one of the
best ways of getting single-machine multicore processing.

Interestingly, the worker threads in OP can probably be made
fork-safe. Not sure that's especially useful, but I can imagine.

-- Devin