subprocess.Popen pipeline bug?

[Douglas Wells]

In article <m38x0mg84i.fsf at elektro.pacujo.net>,
  Marko Rauhamaa <marko at pacujo.net> writes:
> 
> This tiny program hangs:
> 
> ========================================================================
> #!/usr/bin/env python
> import subprocess
> a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
>                      stdout = subprocess.PIPE)
> b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
> a.stdin.close()
> b.wait() # hangs
> a.wait() # never reached
> ========================================================================
> 
> It shouldn't, should it?

Yes, it should.

This issue is related to the subtleties of creating a pipeline in
POSIX environments.  The problem is that the cat command in
subprocess a never completes because it never encounters an EOF
(on a.stdin).  Even though you issue a close call (a.stdin.close ()),
you're not issuing the "last" close.  That's because there is still
at least one file descriptor open in subprocess tree b.  That
happened because it was open when the subprocess module executed
a POSIX fork call and it got duplicated as part of the fork call.

I don't see any clean and simple way to actually fix this.  (That's
one of the reasons why POSIX shells are so complicated.)  There
are a couple of work-arounds that you can use:

1) Force close-on-exec on the specific file descriptor:

    import subprocess
    a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
                         stdout = subprocess.PIPE)
    # ********* beginning of changes
    import os, fcntl
    fd = a.stdin.fileno ()
    old = fcntl.fcntl (fd, fcntl.F_GETFD)
    fcntl.fcntl (fd, fcntl.F_SETFD, old | fcntl.FD_CLOEXEC)
    # ********* end of changes
    b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
    a.stdin.close()
    b.wait()
    a.wait()

Or if it happens to not cause undesired side-effects for you, you can
2) Force close-on-exec *all* non-standard file descriptors by using
the close_fds argument to Popen:

    import subprocess
    a = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE,
                         stdout = subprocess.PIPE)
    # ********* beginning of changes
    # b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout)
    b = subprocess.Popen('cat >/dev/null',shell = True,stdin = a.stdout,
                         close_fds = True)
    # ********* end of changes
    a.stdin.close()
    b.wait()
    a.wait()

Good luck.

 - dmw

-- 
.   Douglas Wells             .  Connection Technologies      .
.   Internet:  -sp9804- -at - contek.com-                     .