os.popen and lengthy operations

[Marc 'BlackJack' Rintsch]

On Thu, 20 Sep 2007 10:31:43 +0400, Dmitry Teslenko wrote:

> I'm using os.popen to perform lengthy operation such as building some
> project from source.
> It looks like this:
> def execute_and_save_output( command, out_file, err_file):
> 
> import os
> 
> def execute_and_save_output( command, out_file, err_file):
> 	(i,o,e) = os.popen3( command )
> 	try:
> 		for line in o:
> 			out_file.write( line )
> 
> 		for line in e:
> 			err_file.write( line )
> 	finally:
> 		i.close()
> 		o.close()
> 		e.close()
> 
> ...
> execute_and_save_output( '<some long to run command>', out_file, err_file)
> 
> Problem is that script hangs on operations that take long to execute
> and have lots of output such as building scripts.

Your code reads from the process' stdout until there is nothin to read
anymore and then from stderr.  The process might output something to both.
The output is buffered.  And when the stderr buffer is full the process
blocks until your application reads something from `e`.  That's where the
whole thing hangs.  You wait for something on `o` and the process waits
for you to read from `e` → deadlock.

You have to use threads to read both `o` and `e` or the `select` module to
look which file has something to read.

Ciao,
	Marc 'BlackJack' Rintsch