Trouble with popen2

[Rembrandt Q Einstein]

Rembrandt Q Einstein wrote:
> I am running an external command and I need to know a) when it is done 
> and b) what it wrote to both stdout and stderr.  After a little 
> searching, I found the popen2 module and used the Popen3 class.  I'm 
> having trouble with it hanging, though.
> 
> Here is a very well put (by someone else) posting that describes some 
> background:
> 
> http://mail.python.org/pipermail/python-list/2004-July/230837.html
> 
> I came to a similar conclusion as that poster and his workaround 
> (independently discovered by me) does do the job he requires.  However, 
> I need to also read stderr, so I made this sample writer and runner:
> 
> -------
> #!/usr/bin/python
> 
> import sys
> 
> for line in range(0, int(sys.argv[1])):
>     print "X" * 120
> 
> print >>sys.stderr, "hi"
> -------
> #!/usr/bin/python
> 
> import popen2
> 
> f = popen2.Popen3("./writer.py 50000", True)
> outs = []
> errs = []
> while (f.poll() == -1):
>     errs += f.childerr.readlines()
>     outs += f.fromchild.readlines()
> -----
> 
> 
> This hangs in the childerr.readlines().  Is it blocking?  If so, why? In 
> any case, how can I be sure to read ALL data from BOTH stderr and stdout 
> and not be in danger of hanging?
> 
> 
> PS: I just found that if I swap the order of the readlines() statements, 
> it works.  But I don't want to use that until I understand why.  I 
> suspect it's a race condition and I don't want to rely on that.

It's possible that when I have child.readlines() first, it consumes all 
50000 output lines and then err.readlines() grabs the "hi" and it just 
exists.  When I have them the other way, err.readlines() blocks (I 
thought readlines() was non-blocking, though) and it just hangs.

My real external program isn't nice enough to order the output like 
that, so here's a more realistic writer:

-------------
#!/usr/bin/python

import sys
import math

for i in range(0, int(sys.argv[1])):
     if math.fmod(i, 100) == 0:
         print >>sys.stderr, "hi"
     print >>sys.stdout, "X" * 120
----

The challenge is to write a program that will run this one in a 
subprocess and read all of both stderr and stdout.