spawning unix command processes which run in parallel and simultaneously

[Matthew Dixon Cowles]

On Sun, 10 Jun 2001 18:23:38 +0800, Alan Tsang <atsang at hk.linkage.net>
wrote:

>HI, I am learning to code with Python and I encounter a technical
>problem.

Hi and welcome.

>I want to spawn a few unix command processes from a python script so
>that I can get the output from the unix command processes and use
>them in python.

>But I don't want to wait for the completion of one process at a time
>and then start another.  I want the python script to start the
>processes simultaneously and monitor whether anyone of them have
>finished and if so, get the output from the finished one.

>It seems to me that the os and popen2 module is useful in this
>respect but I am not sure how to do it.

You can do what you want using popen (maybe popen2) and select. I'll
append a simple example. Unfortunately, with pipes and subprocesses,
once you go beyond a deliberately simple example, things can get
messy. Unless you and -- much harder to control -- your subprocesses
get buffering just right, programs that want to be interactive can
block waiting for input that you think you've sent and waiting for you
to read output that you haven't seen.

Happily, other folks have written code to deal with this nuisance.

The Vaults of Parnassus (www.vex.net/parnassus/) is a very valuable
resource. If you have a look in the Tools/Extensions section there
you'll find several modules inspired by Tcl's Expect (the canonical
program-that-controls-programs). I don't know which of them is best
but there aren't so many that it would be hard to try them out. Tim
O'Malley's event loop that's there looks like it might be of use to
you too.

Regards,
Matt


import random
import os
import select
import string

def getCommand():
  return "sleep %i;echo %s" % \
    (random.randint(0,10),random.choice(string.hexdigits))

def main():
  readPipes=[]
  for count in range(5):
    readPipes.append(os.popen(getCommand()))
  while 1:
    # Could put a timeout here if we had something else to do
    readable,writable,errors=select.select(readPipes,[],[])
    for p in readable:
      print p.read()
      readPipes.remove(p)
      os.wait() # Don't want zombies
    if len(readPipes)==0:
      break

if __name__=="__main__":
  main()