Multiprocessing vs subprocess

[oliver]

With multiprocessing you can take advantage of multi-core processing as it
launches a separate python interpreter process and communicates with it via
shared memory (at least on windows). The big advantage of multiprocessing
module is that the interaction between processes is much richer than
subprocess: subprocess is limited to pipes (unless you build your own
alternate comms mechanism), whereas multiprocessing takes care of
marshalling python objects and transporting them to the other processes.
This can be huge in some applications, like we were using it to run batch
monte carlo simulations where we needed to pass simulation related
datastructures, would have been a pain with subprocess.


On Tue, Mar 12, 2019 at 12:30 PM Schachner, Joseph <
Joseph.Schachner at teledyne.com> wrote:

> Re: " My understanding (so far) is that the tradeoff of using
> multiprocessing is that my manager script can not exit until all the work
> processes it starts finish. If one of the worker scripts locks up, this
> could be problematic. Is there a way to use multiprocessing where processes
> are launched independent of the parent (manager) process?"
>
> I just want to point out that subprocess (which uses the operating system
> to start another processes, to run whatever you say - doesn't have to be
> Python running another script, can be an .exe)  does not have that
> restriction.  The subprocess call can be blocking (the caller waits), or
> not.   If not the caller can do whatever it wants including decide not to
> wait forever, and it can even exit.
>
> Of course, if one of the processes you launched is locked up that is a
> problem all by itself, even thought the manager can exit.
>
> --- Joseph S.
> --
> https://mail.python.org/mailman/listinfo/python-list
>