[New-bugs-announce] [issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits

[Andriy Maletsky]

New submission from Andriy Maletsky <andriy.maletsky at gmail.com>:

When communicate() is called in a loop, it crashes when the child process has already closed any piped standard stream, but still continues to be running.

How this happens:
1) the parent waits for the child events inside communicate() call
2) the child closes its side of any attached pipes long before exiting (in my case there is some complex c++ application which had messed with its termination)
3) communicate() receives an epoll event, tries to read/write, receives SIGPIPE (for stdin) or EOF (for stdout), decides to close corresponding file descriptors from its side
4) communicate() waits for the death of the child, but a timeout is fired
5) parent handles timeout exception and calls communicate() again
6) an exception is raised when communicate() tries to register closed file in epoll

I think there may be a simple solution: before registering file descriptors in epoll, we may check whether any of them is already closed, and don't register it in that case.

Here is a simple reproducible example, ran on Linux 4.15.0-1021-aws x86_64:


import subprocess

child = subprocess.Popen(
    ['/usr/local/bin/python3.7', '-c', 'import os, time; os.close(1), time.sleep(30)'],
    stdout=subprocess.PIPE,
)

while True:
    try:
        child.communicate(timeout=3)
        break
    except subprocess.TimeoutExpired:
        # do something useful here
        pass


Here is a stacktrace:

Traceback (most recent call last):
  File "test.py", line 10, in <module>
    child.communicate(timeout=3)
  File "/usr/local/lib/python3.7/subprocess.py", line 933, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/local/lib/python3.7/subprocess.py", line 1666, in _communicate
    selector.register(self.stdout, selectors.EVENT_READ)
  File "/usr/local/lib/python3.7/selectors.py", line 352, in register
    key = super().register(fileobj, events, data)
  File "/usr/local/lib/python3.7/selectors.py", line 238, in register
    key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
  File "/usr/local/lib/python3.7/selectors.py", line 225, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/local/lib/python3.7/selectors.py", line 40, in _fileobj_to_fd
    "{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=3>

----------
messages: 329412
nosy: and800
priority: normal
severity: normal
status: open
title: Popen.communicate() breaks when child closes its side of pipe but not exits
type: crash
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35182>
_______________________________________