Executing multiple subprocesses and waiting

[Jean-Michel Pichavant]

JB wrote:
> One of my python scripts that takes a bunch of inputs from a tKinter
> gui, generates a set of command line stings, and then threads them off
> to subprocess for calls to other programs like Nuke and our render
> farm has recently started randomly crashing pythonw.exe.
>
> I'm taking a look at my threading setup and attempting to clean it up.
> I was wondering what a smart way of doing what I describe is? Take a
> list of strings containing command line calls to other programs,
> process them one at a time (waiting for the previous one to complete
> before starting the next) and then finishing elegantly. Currently, the
> gui passes them all to a "workerThread" which loops through each
> string, sending it to a "processThread" which makes a call to
> subprocess to execute it.
>
> This has worked fine for over a year so the recent crashing is
> mystifying me. I'm wondering if it's increased network stress (we've
> grown) or something similar?
>
> Any thoughts and suggestions on waiting for threads to complete are
> appreciated.
>   

Google 'Python execnet'.

The main purpose of this module is to execute jobs on a bunch of 
different machines. However by simply declaring your local as unique 
available gateway, execnet will take care of queueing the jobs on your 
local. Look for examples with 'Popen' type gateways.

I love this module, (and any other that are doing the network stuff for 
you).

JM