Alternative to subprocess in order to not wait for calling commands to complete

[Adam Skutt]

On May 10, 12:07 pm, ks <gridsngat... at gmail.com> wrote:
> Hi All,
>
> From within one Python program, I would like to invoke three other
> Python programs. Usually I would use the subprocess module to invoke
> these sequentially. I now have a use case in which I must invoke the
> first one (not wait for it to complete), then invoke the second
> (similarly not wait for it to complete) and then go on to the third.
>
> I am not sure where I should start looking to be able to do this. I am
> reading about threads and forking and there are many options out
> there. So I was wondering if anyone might have suggestions on where I
> can start.
>

subprocess.Popen objects only block for process termination when you
request it, via wait() or communicate().  As such, if you never call
those methods, you will never block for process termination.  What you
want is essentially the default behavior if you're creating Popen
objects directly (instead of using the call() function).

However, you must call the wait() or poll() methods at least once,
after the subprocess has terminated, to release system resources.
There are many ways to accomplish this, and the best approach depends
on your application.  One simple option is to call poll() once each
time your application goes through its main loop.  Another option is
to use a dedicated thread for the purpose of reaping processes.

Please note that calling Popen.poll() in a loop over many processes
does not scale (each call incurs an expensive system call) and will
cause noticeable CPU consumption with even a handful of processes (3
is just fine, though).  Unfortunately, the subprocess module lacks any
way to wait on multiple processes simultaneously.  This is possible
with operating-system specific code, however.  I would not worry about
it for now, but it may be something you need to consider in the
future.

Adam