Python 3: Launch multiple commands(subprocesses) in parallel (but upto 4 any time at same time) AND store each of their outputs into a variable

[Rob Gaddi]

lax.clarke at gmail.com wrote:

> Hi,
>
> I've been reading various forums and python documentation on subprocess, multithreading, PIPEs, etc.  But I cannot seem to mash together several of my requirements into working code.
>
> I am trying to:
>
> 1) Use Python 3+ (specifically 3.4 if it matters)
> 2) Launch N commands in background (e.g., like subprocess.call would for individual commands)
> 3) But only limit P commands to run at same time
> 4) Wait until all N commands are done
> 5) Have an array of N strings with the stdout+stderr of each command in it.
>
> What is the best way to do this?
> There are literally many variations of things in the Python documentation and Stackoverflow that I am unable to see the forest from trees (for my problem).
>
> Thank you very much!

First off, I'm assuming that the stdout+stderr of these commands is of
reasonable size rather than hundreds of megabytes.

What you want is a finite pool of threads (or processes) that execute
the tasks.  multiprocessing.pool.Pool will do it.  So will
concurrent.futures, which is what I'd personally use just out of more
familiarity with it.

In either case your task should wrap a call to subprocess. 
subprocess.run is your easiest answer if you've got Python 3.5; the task
would call it with stdout and stderr=subprocess.PIPE, get the
CompletedProcess back, and then store the .stdout and .stderr string
results.  For older Python, create a subprocess.Popen (again with stdout
and stderr=subprocess.PIPE) and call the communicate() method.

There's probably a dozen other ways.  That one there, that's your
easiest.

-- 
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order.  See above to fix.