Multiprocessing vs subprocess to run Python scripts in parallel

[Chris Angelico]

On Wed, Mar 13, 2019 at 2:01 AM Malcolm Greene <python at bdurham.com> wrote:
>
> Use case: I have a Python manager script that monitors several conditions (not just time based schedules) that needs to launch Python worker scripts to respond to the conditions it detects. Several of these worker scripts may end up running in parallel. There are no dependencies between individual worker scripts. I'm looking for the pros and cons of using multiprocessing or subprocess to launch these worker scripts. Looking for a solution that works across Windows and Linux. Open to using a 3rd party library. Hoping to avoid the use of yet another system component like Celery if possible and rational.
>
> My understanding (so far) is that the tradeoff of using multiprocessing is that my manager script can not exit until all the work processes it starts finish. If one of the worker scripts locks up, this could be problematic. Is there a way to use multiprocessing where processes are launched independent of the parent (manager) process?
>

One advantage of multiprocessing is that it can fork, rather than
spinning up an entire new Python process from scratch. This doesn't
work on Windows, but it can give huge performance advantages on Unix
systems.

If the subprocesses are basically spun off and abandoned, you probably
want to use subprocess, but at this point, you're talking
architectural style and there are no right/wrong answers, just
consequences of different decisions. For instance, multiprocessing
will always run all the children in the exact same Python interpreter
that the master was invoked from, but with subprocess, you can invoke
anything you like (doesn't even have to be Python). OTOH, the
aforementioned performance advantage could be pretty significant if
you're starting lots of different processes in a short space of time.
And maybe you don't even need subprocesses at all, but could do it all
with threads or other forms of asynchronicity. You may need to mess
around a bit before making a decision.

ChrisA