[issue35182] Popen.communicate() breaks when child closes its side of pipe but not exits
Andriy Maletsky
report at bugs.python.org
Wed Nov 7 05:23:30 EST 2018
New submission from Andriy Maletsky <andriy.maletsky at gmail.com>:
When communicate() is called in a loop, it crashes when the child process has already closed any piped standard stream, but still continues to be running.
How this happens:
1) the parent waits for the child events inside communicate() call
2) the child closes its side of any attached pipes long before exiting (in my case there is some complex c++ application which had messed with its termination)
3) communicate() receives an epoll event, tries to read/write, receives SIGPIPE (for stdin) or EOF (for stdout), decides to close corresponding file descriptors from its side
4) communicate() waits for the death of the child, but a timeout is fired
5) parent handles timeout exception and calls communicate() again
6) an exception is raised when communicate() tries to register closed file in epoll
I think there may be a simple solution: before registering file descriptors in epoll, we may check whether any of them is already closed, and don't register it in that case.
Here is a simple reproducible example, ran on Linux 4.15.0-1021-aws x86_64:
import subprocess
child = subprocess.Popen(
['/usr/local/bin/python3.7', '-c', 'import os, time; os.close(1), time.sleep(30)'],
stdout=subprocess.PIPE,
)
while True:
try:
child.communicate(timeout=3)
break
except subprocess.TimeoutExpired:
# do something useful here
pass
Here is a stacktrace:
Traceback (most recent call last):
File "test.py", line 10, in <module>
child.communicate(timeout=3)
File "/usr/local/lib/python3.7/subprocess.py", line 933, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/local/lib/python3.7/subprocess.py", line 1666, in _communicate
selector.register(self.stdout, selectors.EVENT_READ)
File "/usr/local/lib/python3.7/selectors.py", line 352, in register
key = super().register(fileobj, events, data)
File "/usr/local/lib/python3.7/selectors.py", line 238, in register
key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
File "/usr/local/lib/python3.7/selectors.py", line 225, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/local/lib/python3.7/selectors.py", line 40, in _fileobj_to_fd
"{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=3>
----------
messages: 329412
nosy: and800
priority: normal
severity: normal
status: open
title: Popen.communicate() breaks when child closes its side of pipe but not exits
type: crash
versions: Python 2.7, Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35182>
_______________________________________
More information about the Python-bugs-list
mailing list