Trouble with popen2

[Rembrandt Q Einstein]

Rembrandt Q Einstein wrote:
> Rembrandt Q Einstein wrote:
> 
>> I am running an external command and I need to know a) when it is done 
>> and b) what it wrote to both stdout and stderr.  After a little 
>> searching, I found the popen2 module and used the Popen3 class.  I'm 
>> having trouble with it hanging, though.
>>
>> Here is a very well put (by someone else) posting that describes some 
>> background:
>>
>> http://mail.python.org/pipermail/python-list/2004-July/230837.html
>>
>> I came to a similar conclusion as that poster and his workaround 
>> (independently discovered by me) does do the job he requires.  
>> However, I need to also read stderr, so I made this sample writer and 
>> runner:
>>
>> -------
>> #!/usr/bin/python
>>
>> import sys
>>
>> for line in range(0, int(sys.argv[1])):
>>     print "X" * 120
>>
>> print >>sys.stderr, "hi"
>> -------
>> #!/usr/bin/python
>>
>> import popen2
>>
>> f = popen2.Popen3("./writer.py 50000", True)
>> outs = []
>> errs = []
>> while (f.poll() == -1):
>>     errs += f.childerr.readlines()
>>     outs += f.fromchild.readlines()
>> -----
>>
>>
>> This hangs in the childerr.readlines().  Is it blocking?  If so, why? 
>> In any case, how can I be sure to read ALL data from BOTH stderr and 
>> stdout and not be in danger of hanging?
>>
>>
>> PS: I just found that if I swap the order of the readlines() 
>> statements, it works.  But I don't want to use that until I understand 
>> why.  I suspect it's a race condition and I don't want to rely on that.
> 
> 
> It's possible that when I have child.readlines() first, it consumes all 
> 50000 output lines and then err.readlines() grabs the "hi" and it just 
> exists.  When I have them the other way, err.readlines() blocks (I 
> thought readlines() was non-blocking, though) and it just hangs.
> 
> My real external program isn't nice enough to order the output like 
> that, so here's a more realistic writer:
> 
> -------------
> #!/usr/bin/python
> 
> import sys
> import math
> 
> for i in range(0, int(sys.argv[1])):
>     if math.fmod(i, 100) == 0:
>         print >>sys.stderr, "hi"
>     print >>sys.stdout, "X" * 120
> ----
> 
> The challenge is to write a program that will run this one in a 
> subprocess and read all of both stderr and stdout.

Ah--despite my new nick, I am an idiot.  I should just use select() on 
fromchild and childerr.  Something like this:

while f.poll() == -1:
   select([], [f.fromchild, f.childerr],[], 0)
     blah

However, I think I might still miss some data this way.