Using subprocess module to launch a shell shell script that itself forks a process

[Gabriel Genellina]

En Wed, 08 Oct 2008 15:24:39 -0300, Samuel A. Falvo II  
<sam.falvo at gmail.com> escribió:
> On Oct 7, 6:23 pm, "Gabriel Genellina" <gagsl-... at yahoo.com.ar> wrote:

>> you set stdin=PIPE - is your java process expecting some input? you're  
>> not writing anything to stdin.
>
> It does not expect input from stdin.  However, this does not affect
> any OTHER scripts or commands I run.

But it *does* affect how subprocess handles the child process internally.

> Let's remember to look at the objective facts: for shell scripts that
> launch child processes of their own, Python hangs.  For all other
> types of commands, it works 100% as expected.

Don't conclude too fast... I've tested a variant (using a Perl script as a  
replacement instead of a Java program), and it worked fine:

<log>
gabriel at sleipnir:~$ cat foo.sh
#!/bin/sh
perl foo.pl&
echo End of foo.sh

gabriel at sleipnir:~$ cat foo.pl
#!/usr/bin/perl

for ($i=5; $i>0; $i--) {
   print "$i seconds remaining...\n";
   sleep(1);
}
print "End of foo.pl\n";

gabriel at sleipnir:~$ cat foo.py
#!/usr/bin/python2.5

import subprocess

command = "./foo.sh"

p = subprocess.Popen(command,
         shell=True,
         stdin=None,
         stdout=subprocess.PIPE,
         stderr=subprocess.STDOUT,
         close_fds=True
)
print "py: before p.communicate"
output = p.communicate()[0]
print "py: after p.communicate"
print "py: returncode=",p.returncode
print "py: read",len(output),"bytes"
print "py: output=", repr(output)
print "py: End of foo.py"


gabriel at sleipnir:~$ ./foo.py
py: before p.communicate
py: after p.communicate
py: returncode= 0
py: read 143 bytes
py: output= 'End of foo.sh\n5 seconds remaining...\n4 seconds  
remaining...\n3 seconds remaining...\n2 seconds remaining...\n1 seconds  
remaining...\nEnd of foo.pl\n'
py: End of foo.py
gabriel at sleipnir:~$ uname -a
Linux debian 2.6.18-4-486 #1 Mon Mar 26 16:39:10 UTC 2007 i686 GNU/Linux
gabriel at sleipnir:~$
</log>

This was tested both with python 2.4.4 and 2.5.2
I don't think it's a Python problem, but something related to java and how  
it handles stdin/stdout. Try with another program.

> But, my question now is, WHY is this an issue?  If the launched
> process doesn't read from its stdin, why would it block?

Several reasons - the child process might send enough text to stderr to  
fill its buffer, and if the parent process doesn't read from the other end  
of the pipe, the child blocks. That's why I suggested to use communicate  
instead of stdout.read() - it takes care of such cases.

> The only thing I can think of is that the JVM's stdout is tied to the
> shell script's stdout, because that's the only way it can remain open
> upon the termination of the child process.  I suppose, at this point,
> the next place to look is in shell script syntax to find out how to
> detach the JVM from the script's process group.

Can't you redirect to a file instead?
java foo.jar  >/tmp/foo.log 2>&1 &

-- 
Gabriel Genellina