Using subprocess module to launch a shell shell script that itself forks a process

[Derek Martin]

On Tue, Oct 07, 2008 at 05:43:41PM -0700, Samuel A. Falvo II wrote:
>     p = subprocess.Popen(
>         command,
>         shell=True,
>         stdin=subprocess.PIPE,
>         stdout=subprocess.PIPE,
>         stderr=subprocess.STDOUT,
>         close_fds=True
>     )
> 
>     outputChannel = p.stdout
>     output = outputChannel.read()

You call read with no arguments.  This is highly problematic in the
context of interprocess communication, unless you can be 100% positive
that none of the children will write anywhere besides STDOUT, and
won't try to read from STDIN in the meanwhile.

Python's read() with no args reads until the end of file, which in IPC
contexts is bad...  Normally the child process won't close stdout
until it exits.  So, if it did any other output in between, say, to
STDERR, the child will block waiting for the parent to read STDERR,
meanwhile the parent is blocked waiting for input from the child's
STDOUT, which results in a deadlock.  Both processes sleep forever.
The exact same thing can happen if either the shell script or a
process started by the shell script tries to read from STDIN.

Since Java is launched by the shell script, it inherits the shell
script's STDIN, STDOUT, and STDERR file descriptors (i.e. the two
processes share the same STDIO).  Thus if the java process writes to
STDERR, that also could be causing your deadlock.


On Wed, Oct 08, 2008 at 11:24:39AM -0700, Samuel A. Falvo II wrote:
> On Oct 7, 6:23 pm, "Gabriel Genellina" <gagsl-... at yahoo.com.ar> wrote:
> > Is your shell script doing something else, apart from invoking the java  
> > process?
> 
> Obviously, yes.  

It's far from obvious.  I can't count the number of scripts I've seen
whose sole purpose was to launch a Java program (with the right
environment)...  You would do well to avoid being dismissive and
listen to those who respond with help,  when you are the one who
obviously doesn't understand the behavior you're getting, and you're
the one asking for help.

> The script is some 150 lines long.  But the hang-up
> occurs because of the forked Java process, not the other lines.

I'm betting it's because the Java program is writing warnings to
STDERR (more data than will fit in the STDIO buffer), which you're not
reading...

> > If not, you could just invoke java directly from Python. Also,  
> > you set stdin=PIPE - is your java process expecting some input? you're not  
> > writing anything to stdin.
> 
> It does not expect input from stdin.  However, this does not affect
> any OTHER scripts or commands I run.

Irrelevant...  Unless "any OTHER scripts" encompases all possible
combinations of process interactions, and you can demontstrate that it
does so, this proves nothing.

> Let's remember to look at the objective facts: for shell scripts that
> launch child processes of their own, Python hangs.  For all other
> types of commands, it works 100% as expected.

These are not facts which are in evidence.  We don't know what your
script is doing, and it's clear that you yourself are not able to
explain the behavior you are seeing, therefore there is no reason for
us to conclude that the above statements are true and correct.  Most
likely, they are not.

> > Anyway, it's better to use the communicate method instead (it uses select  
> > to read from both stdout and stderr):
> 
> That doesn't help me.

Please explain why it doesn't.

The most likely cause of the behavior you are seeing is the deadlock I
described, above.   Using select() (i.e. using communicate()) should
generally fix about half the cases...  The rest would be fixed by
redirecting STDIN of the child (or at least the Java process) from
/dev/null, most likely.

Then of course, there could be other explanations.  But this being
overwhelmingly the most likely one, if you don't try those solutions,
there's no point in trying to help you further...

-- 
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-list/attachments/20081009/5d6673fd/attachment-0001.sig>