spawning a process with subprocess

[bhunter]

On Nov 26, 3:05 pm, "Diez B. Roggisch" <de... at nospam.web.de> wrote:
> bhunter schrieb:
>
>
>
> > On Nov 26, 1:50 pm, "Diez B. Roggisch" <de... at nospam.web.de> wrote:
> >> bhunter schrieb:
>
> >>> Hi,
> >>> I've used subprocess with 2.4 several times to execute a process, wait
> >>> for it to finish, and then look at its output.  Now I want to spawn
> >>> the process separately, later check to see if it's finished, and if it
> >>> is look at its output.  I may want to send a signal at some point to
> >>> kill the process.  This seems straightforward, but it doesn't seem to
> >>> be working.
> >>> Here's my test case:
> >>> import subprocess, time
> >>> cmd = "cat somefile"
> >>> thread = subprocess.Popen(args=cmd.split(), shell=True,
> >>> stdout=subprocess.PIPE, stdin=subprocess.PIPE,
> >>> stderr=subprocess.STDOUT, close_fds=True)
> >>> while(1):
> >>>       time.sleep(1)
> >>>       if(thread.returncode):
> >>>          break
> >>>       else:
> >>>          print thread.returncode
> >>> print "returncode = ", thread.returncode
> >>> for line in thread.stdout:
> >>>    print "stdout:\t",line
> >>> This will just print the returncode of None forever until I Ctrl-C it.
> >>> Of course, the program works fine if I call thread.communicate(), but
> >>> since this waits for the process to finish, that's not what I want.
> >>> Any help would be appreciated.
> >> I have difficulties understanding what you are after here. To me it
> >> looks as if everything works as expected. I mean you periodically check
> >> on the liveness of the "thread" - which is what you describe above. All
> >> you are missing IMHO is the actual work in this program.
>
> >> So
>
> >> while True:
> >>      if do_work():
> >>          if thread.returncode:
> >>              break
> >>      else:
> >>          thread.kill()
>
> >> This assumes that your do_work()-method communicates the wish to end the
> >> sub-process using it's returnvalue.
>
> >> Diez
>
> > If the subprocess had finished, I expect that the returncode will not
> > be None, and the loop would break.  The process hasn't actually
> > started.  I know this because while this simple testcase just cats a
> > file, the real case submits a simulation job.  This job never starts
> > until after I ctrl-c the program.
>
> I don't know what the reason is for that, but I've just today worked
> with code that exactly uses subprocess as advertised - spawning a
> process which runs while the main process occasionally checks inside the
> child's logfiles for certain state changes.
>
> What might be though is that you need to consume the subprocesses stdout
> in your program - because otherwise it will buffer until a certain
> amount (usually 4 or 16k) and then halt.
>
> Diez

It works?  You mean there are place on earth where it really does
this?  Excellent!

Still doesn't work for me, though.  I first tried changing bufsize to
-1, then 0, then a very large number greater than the number of bytes
in this file (just to be sure).  None seemed to have any affect.

Then I modified the code to this:

cmd = "cat /nfs/dv1/bhunter/o52a/verif/gmx/rgx.cc"
args = cmd.split()
thread = subprocess.Popen(args=args, shell=True,
stdout=subprocess.PIPE, stdin=subprocess.PIPE,
stderr=subprocess.STDOUT, close_fds=True, bufsize=300000)

lines = []

try:
   while(1):
      time.sleep(1)
      if(thread.returncode):
         break
      else:
         print thread.returncode
      lines.extend(thread.stdout.readlines())
except KeyboardInterrupt:
   print lines

print "returncode = ", thread.returncode
for line in thread.stdout:
   print "stdout:\t",line


This one hangs after the first print of returncode None.  Then, I ctrl-
C it and find that the lines array is empty.

It's my guess that the process is waiting for some input, even though
'cat'  clearly does not require anything from stdin.  But if I put a
communicate() after Popen and a few other tweaks, then everything
works as expected--but of course not as desired.

Still confused.

Brian