os.popen3 hangs in Windows XP SP1, SP2. Python 2.5 & 2.4. Consistent test case.

[Pierre Rouleau]

On Dec 12, 10:11 am, Thomas Guettler
<guettli.use... at thomas-guettler.de> wrote:
> Pierre Rouleau wrote:
> > Hi all,
>
> > I have a consistent test case where os.popen3() hangs in Windows.  The
> > system hangs when retrieving the lines from the child process stdout.
> > I know there were several reports related to os.popen3() hanging under
> > Windows in this group before.I had a problem like this some time ago. But it was a problem that
> would happen under any operating system.
>
> If there is output on stdout and stderr, you will get a dead lock sooner
> or later.
>
> Example: The childprocess tries to write to stderr, and the
> parent process reads from stdin. The buffer of stderr will
> get full. The child will block. The parent will wait for ever.
>
> Seehttp://docs.python.org/lib/popen2-flow-control.html
>
> My hint: Always use popen4
>
> You can get dead locks with popen4, too. But only if you
> write to pipe.tochild.

Thanks for replying Thomas.

The reason I was using popen3() is that I need to parse the error
stream and do something else with the stdout.   My operation does not
need to have concurrent operations of the parent and child.  I could
have used something file::

os.system('parent > log_stdout.txt  2> log_stderr.txt')

and then parse the 2 files.  I just wanted to avoid using the files to
avoid having to have to deal with issues related to temporary file
names if there where several process instance of the code running
simultaneously.


Now, from the reading of the above link, this means that If I want to
be able to do what I want with pipes, avoiding deadlock means that:

- In the parent program, the code should be something that looks like:

    stdin, stdout, stderr = os.popen3(command)
    stderr_value = list(stderr)
    stdout_value = list(stdout)
    pgm_exit_code = stdout.close() or 0
    stdin.close()
    stderr.close()

An the above would work only if one stream is written by the child at a
time and stderr closed:

    for whatever: print >> sys.stderr, ' the error messages'
    os.close(sys.stderr.fileno())
    for someother: print 'other stdout info'

In my case, since I don't control the child program, I can assume that
it does not follow the required order. I am launching nosetests which
runs other test programs.  I tried closing sys.stderr in the teardown
of my test script and that removed the deadlock but caused other
problems (because sys.stderr is used later by nosetests).

So, in the end, it looks like I really don't have any choice: if I want
to safely read both stdout and stderr in a way that is child program
agnostic: I must use temporary files (or maybe use something like
select).  Right?

Thanks again!

--

Pierre Rouleau