[issue22114] You cannot call communicate() safely after receiving EAGAIN
Amrith Kumar
report at bugs.python.org
Thu Jul 31 13:58:59 CEST 2014
New submission from Amrith Kumar:
Environment:
- Linux (Ubuntu 14.04 LTS, x64) running Python 2.7.
- Code uses eventlet.green.subprocess
Code establishes subprocess as:
subprocess.Popen(command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE, ...)
It then calls communicate()
communicate() throws EAGAIN
The code calls communicate() again
This fails with "ValueError: I/O operation on closed file"
This exception comes from eventlet/greenio.py because an operation (flush) has been attempted on a closed file.
The complete explanation of this failure is this.
When communicate() is called in this way (WITH NO INPUT but with all three handles stdin, stdout, stderr), code in communicate() bypasses the "1 handle optimization" and goes directly to _communicate()
_communicate() will first send any input on stdin along, and flush and close stdin().
>From that point forward, if anything to do with stderr and stdout returns EAGAIN, an attempt to call communicate() again will fail because stdin has now been closed().
This is because the code in _communicate() preemptively closes stdin if there is no input but does not reset stdin.
def _communicate(self, input):
if self.stdin:
# Flush stdio buffer. This might block, if the user has
# been writing to .stdin in an uncontrolled fashion.
self.stdin.flush()
if not input:
self.stdin.close()
The fix for this is relatively straightforward (conceptually).
def _communicate(self, input):
if self.stdin:
# Flush stdio buffer. This might block, if the user has
# been writing to .stdin in an uncontrolled fashion.
self.stdin.flush()
if not input:
self.stdin.close()
self.stdin = None # This should take care of it.
However, a partial workaround from the client perspective is this.
1. If you have no input, set stdin to None
2. If you do have input and you get EAGAIN, you cannot safely retry because your input may have already been completely or partially sent to the subprocess; you have to assume failure.
--------------------
Backtrace:
Traceback (most recent call last):
File "/opt/stack/trove/trove/tests/unittests/guestagent/test_mongodb_manager.py", line 58, in test_prepare_from_backup
self._prepare_dynamic(backup_id='backup_id_123abc')
File "/opt/stack/trove/trove/tests/unittests/guestagent/test_mongodb_manager.py", line 91, in _prepare_dynamic
backup_info=backup_info)
File "trove/guestagent/datastore/mongodb/manager.py", line 66, in prepare
operating_system.update_owner('mongodb', 'mongodb', mount_point)
File "trove/guestagent/common/operating_system.py", line 109, in update_owner
run_as_root=True, root_helper="sudo")
File "trove/common/utils.py", line 278, in execute_with_timeout
return execute(*args, **kwargs)
File "trove/openstack/common/processutils.py", line 186, in execute
result = obj.communicate()
File "/usr/lib/python2.7/subprocess.py", line 799, in communicate
return self._communicate(input)
File "/usr/lib/python2.7/subprocess.py", line 1396, in _communicate
self.stdin.flush()
File "/opt/stack/trove/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenio.py", line 419, in _operationOnClosedFile
raise ValueError("I/O operation on closed file")
ValueError: I/O operation on closed file
----------
components: IO
messages: 224398
nosy: amrith
priority: normal
severity: normal
status: open
title: You cannot call communicate() safely after receiving EAGAIN
versions: Python 2.7
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue22114>
_______________________________________
More information about the Python-bugs-list
mailing list