Python 3: Launch multiple commands(subprocesses) in parallel (but upto 4 any time at same time) AND store each of their outputs into a variable

[Dale Marvin]

On 8/23/16 8:15 PM, lax.clarke at gmail.com wrote:

 > I am trying to:
 >
 > 1) Use Python 3+ (specifically 3.4 if it matters)
 > 2) Launch N commands in background (e.g., like subprocess.call would 
for individual commands)
 > 3) But only limit P commands to run at same time
 > 4) Wait until all N commands are done
 > 5) Have an array of N strings with the stdout+stderr of each command 
in it.
 >
 > What is the best way to do this?

The best way is a matter of opinion, I have had success using Celery 
with Redis. <http://www.celeryproject.org/>

DAle