popen2.popen2,3,4 from Python 2.0 on NT 4.0 SP5

[David Bolen]

Mark Tompkins <mdtompkins at home.com> writes:

> I was doing some testing, and have hit the wall right off the bat.  When
> I use popen2.popenX('cmd') to call the sqlplus utility, reading stdout
> of the cmd appears to hang when the read is past the last character in
> the buffer, or when there is no data (as what happens, if there is an
> error)????

Using normal file I/O on the pipe to the child stdout will typically
block if the handle is still valid (e.g., it hasn't been closed by the
client) but there is currently no data.

>From your problem, it sounds like it may have something to do with how your
sqlplus utility deals with its file handles, particularly in an error
condition.

> import popen2
> 
> # it doesn't matter which popen function? that I use, only difference is
> how failure occurs
> # (whether stdout and stderr are mixed or not)
> 
> x=popen2.popen3('sqlplus pf/pf at dds230pf3 @rowcount.sql')
> y=x[0].readline()
> 
> # if rowcount.sql doesn't exist, hang occurs here immediately

A question would be what is sqlplus doing with that error.  If, for
example, it's writing an error message to stderr, and then flushing
that before exiting, you could be in a deadlock situation.  Until you
read the data from stderr, your sqlplus utility isn't going to exit
and close stdout, and until stdout is closed, your readline() is going
to block waiting for more data.

Alternatively, the sqlplus utility could be waiting for stdin to close
for some reason - if you aren't going to be feeding input to the child
process that way, I'd suggest closing that handle, or not using a
popen# that gives it to you.  Unless you've found that sqlplus doesn't
run without a valid stdin.

I (like Mark's post) find that if I actually have an invalid command,
that the readline() returns immediately with the empty string.  And
for me this is under NT4, not Win2K.  But actually failing to start
the sqlplus utility (since I don't have it) is different from starting
it, but the utility itself deciding there's an error, since that may
control how it uses its output streams.

You might try just using a normal os.popen and adding a "2>&1" to the
end of the command - if only to see if the behavior varies.  This will
use stdout for both normal output and perhaps the error and should
avoid the deadlock.  You could then check for the exit code of the
sqlplus utility to see if there was an error.  If you need a popen#
that maintains stdin to make sqlplus work, then you could still try
redirecting the stderr and just use popen2.

> x[0] is the stdout of the shelled command, which is where I read from
> for input to my program -> file in read mode.
> 
> 1.  Why does it hang?  This seems like a bug.  It should return with
> some known response, or nothing.

Not if the handle is still a valid link to the child but there is no
data pending.  In that case, the I/O request to the pipe to the child
is just going to block waiting for something new to arrive from the
child.  It's the nature of the pipe to the child process.  If my guess
is right, you've just managed to create a case where your script and
the utility are both waiting for each other but on the opposite
stream.

> 2.  Is there a way to find out how many characters are left in the
> stdout/stderr buffers?

I'm not sure that there's any portable way to do this.  I suppose
under Unix you could select on the filesystem descriptor for the
pipes, but select under NT only handles sockets.  Under NT a more
native way would be to set up an overlapped I/O, but that's getting
pretty far afield from Python (I'm not even positive you could do it).

> 3.  This test is for winnt.  Is unix different, and should I use
> win32all to workaround (I would rather not, if possible)?

I don't think you'll see any difference with win32all, especially
since you're using Python 2.0 - the popen# code is pretty much
identical to win32all when being run under Windows.

> 4.  Should I use threading, and use another thread to see when the
> readline() fails to return (messy)?

Well, not to see when it fails to return, but if you really need
stdout and stderr distinct, then having two threads to be able to
drain the two streams simultaneously may be one answer.  It should at
least let you avoid the potential deadlock of you and your child
process blocking on different streams.

> 5.  Is this a shelled process state problem, or a buffer state problem?

Not sure.

> 6.  Does this problem throw any exception?

It shouldn't.

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/