[Python-checkins] cpython (3.4): asyncio doc: rewrite subprocess doc
victor.stinner
python-checkins at python.org
Tue Oct 14 00:53:35 CEST 2014
https://hg.python.org/cpython/rev/d894aa49d5e4
changeset: 93048:d894aa49d5e4
branch: 3.4
user: Victor Stinner <victor.stinner at gmail.com>
date: Tue Oct 14 00:52:07 2014 +0200
summary:
asyncio doc: rewrite subprocess doc
* add a new example using transport and protocol
* rewrite the example using streams to make it much simpler (remove error
handling, use a simpler Python code)
* copy (and adapt) more documentation from the subprocess module:
- add a note about Process.wait() deadlock
- add a note about shell injection
- etc.
* sort Process methods and attributes in the same order than subprocess.Popen
methods and attributes, so the documentation looks closer
* list differences between Process and subprocess.Popen APIs
files:
Doc/library/asyncio-subprocess.rst | 254 +++++++++++-----
Doc/library/subprocess.rst | 2 +-
2 files changed, 179 insertions(+), 77 deletions(-)
diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -27,23 +27,34 @@
Create a subprocess: high-level API using Process
-------------------------------------------------
-.. function:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds)
+.. function:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds)
- Run the shell command *cmd*. See :meth:`BaseEventLoop.subprocess_shell` for
- parameters. Return a :class:`~asyncio.subprocess.Process` instance.
+ Create a subprocess.
- The optional *limit* parameter sets the buffer limit passed to the
- :class:`StreamReader`.
+ The *limit* parameter sets the buffer limit passed to the
+ :class:`StreamReader`. See :meth:`BaseEventLoop.subprocess_exec` for other
+ parameters.
+
+ Return a :class:`~asyncio.subprocess.Process` instance.
This function is a :ref:`coroutine <coroutine>`.
-.. function:: create_subprocess_exec(\*args, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds)
+.. function:: create_subprocess_shell(cmd, stdin=None, stdout=None, stderr=None, loop=None, limit=None, \*\*kwds)
- Create a subprocess. See :meth:`BaseEventLoop.subprocess_exec` for
- parameters. Return a :class:`~asyncio.subprocess.Process` instance.
+ Run the shell command *cmd*.
- The optional *limit* parameter sets the buffer limit passed to the
- :class:`StreamReader`.
+ The *limit* parameter sets the buffer limit passed to the
+ :class:`StreamReader`. See :meth:`BaseEventLoop.subprocess_shell` for other
+ parameters.
+
+ Return a :class:`~asyncio.subprocess.Process` instance.
+
+ It is the application's responsibility to ensure that all whitespace and
+ metacharacters are quoted appropriately to avoid `shell injection
+ <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
+ vulnerabilities. The :func:`shlex.quote` function can be used to properly
+ escape whitespace and shell metacharacters in strings that are going to be
+ used to construct shell commands.
This function is a :ref:`coroutine <coroutine>`.
@@ -121,10 +132,15 @@
Returns a pair of ``(transport, protocol)``, where *transport* is an
instance of :class:`BaseSubprocessTransport`.
+ It is the application's responsibility to ensure that all whitespace and
+ metacharacters are quoted appropriately to avoid `shell injection
+ <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_
+ vulnerabilities. The :func:`shlex.quote` function can be used to properly
+ escape whitespace and shell metacharacters in strings that are going to be
+ used to construct shell commands.
+
This method is a :ref:`coroutine <coroutine>`.
- See the constructor of the :class:`subprocess.Popen` class for parameters.
-
.. seealso::
The :meth:`BaseEventLoop.connect_read_pipe` and
@@ -159,35 +175,37 @@
.. class:: asyncio.subprocess.Process
- .. attribute:: pid
+ A subprocess created by the :func:`create_subprocess_exec` or the
+ :func:`create_subprocess_shell` function.
- The identifier of the process.
+ The API of the :class:`~asyncio.subprocess.Process` class was designed to be
+ closed the API of the :class:`subprocess.Popen` class, but they are some
+ differences:
- Note that if you set the *shell* argument to ``True``, this is the
- process identifier of the spawned shell.
+ * There is no explicit :meth:`~subprocess.Popen.poll` method
+ * The :meth:`~subprocess.Popen.communicate` and
+ :meth:`~subprocess.Popen.wait` methods don't take a *timeout* parameter:
+ use the :func:`wait_for` function
+ * The *universal_newlines* parameter is not supported (only bytes strings
+ are supported)
+ * The :meth:`~asyncio.subprocess.Process.wait` method of
+ the :class:`~asyncio.subprocess.Process` class is asynchronous whereas the
+ :meth:`~subprocess.Popen.wait` method of the :class:`~subprocess.Popen`
+ class is implemented as a busy loop.
- .. attribute:: returncode
+ .. method:: wait()
- Return code of the process when it exited. A ``None`` value indicates
- that the process has not terminated yet.
+ Wait for child process to terminate. Set and return :attr:`returncode`
+ attribute.
- A negative value ``-N`` indicates that the child was terminated by signal
- ``N`` (Unix only).
+ This method is a :ref:`coroutine <coroutine>`.
- .. attribute:: stdin
+ .. note::
- Standard input stream (write), ``None`` if the process was created with
- ``stdin=None``.
-
- .. attribute:: stdout
-
- Standard output stream (read), ``None`` if the process was created with
- ``stdout=None``.
-
- .. attribute:: stderr
-
- Standard error stream (read), ``None`` if the process was created with
- ``stderr=None``.
+ This will deadlock when using ``stdout=PIPE`` or ``stderr=PIPE`` and
+ the child process generates enough output to a pipe such that it
+ blocks waiting for the OS pipe buffer to accept more data. Use the
+ :meth:`communicate` method when using pipes to avoid that.
.. method:: communicate(input=None)
@@ -197,33 +215,28 @@
process, or ``None``, if no data should be sent to the child. The type
of *input* must be bytes.
+ :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``.
+
If a :exc:`BrokenPipeError` or :exc:`ConnectionResetError` exception is
raised when writing *input* into stdin, the exception is ignored. It
occurs when the process exits before all data are written into stdin.
- :meth:`communicate` returns a tuple ``(stdoutdata, stderrdata)``.
-
Note that if you want to send data to the process's stdin, you need to
create the Process object with ``stdin=PIPE``. Similarly, to get anything
other than ``None`` in the result tuple, you need to give ``stdout=PIPE``
and/or ``stderr=PIPE`` too.
+ This method is a :ref:`coroutine <coroutine>`.
+
.. note::
The data read is buffered in memory, so do not use this method if the
data size is large or unlimited.
- This method is a :ref:`coroutine <coroutine>`.
-
.. versionchanged:: 3.4.2
The method now ignores :exc:`BrokenPipeError` and
:exc:`ConnectionResetError`.
- .. method:: kill()
-
- Kills the child. On Posix OSs the function sends :py:data:`SIGKILL` to
- the child. On Windows :meth:`kill` is an alias for :meth:`terminate`.
-
.. method:: send_signal(signal)
Sends the signal *signal* to the child process.
@@ -241,53 +254,142 @@
to the child. On Windows the Win32 API function
:c:func:`TerminateProcess` is called to stop the child.
- .. method:: wait():
+ .. method:: kill()
- Wait for child process to terminate. Set and return :attr:`returncode`
- attribute.
+ Kills the child. On Posix OSs the function sends :py:data:`SIGKILL` to
+ the child. On Windows :meth:`kill` is an alias for :meth:`terminate`.
- This method is a :ref:`coroutine <coroutine>`.
+ .. attribute:: stdin
+ Standard input stream (:class:`StreamWriter`), ``None`` if the process
+ was created with ``stdin=None``.
-Example
--------
+ .. attribute:: stdout
-Implement a function similar to :func:`subprocess.getstatusoutput`, except that
-it does not use a shell. Get the output of the "python -m platform" command and
-display the output::
+ Standard output stream (:class:`StreamReader`), ``None`` if the process
+ was created with ``stdout=None``.
+
+ .. attribute:: stderr
+
+ Standard error stream (:class:`StreamReader`), ``None`` if the process
+ was created with ``stderr=None``.
+
+ .. warning::
+
+ Use the :meth:`communicate` method rather than :attr:`.stdin.write
+ <stdin>`, :attr:`.stdout.read <stdout>` or :attr:`.stderr.read <stderr>`
+ to avoid deadlocks due to streams pausing reading or writing and blocking
+ the child process.
+
+ .. attribute:: pid
+
+ The identifier of the process.
+
+ Note that for processes created by the :func:`create_subprocess_shell`
+ function, this attribute is the process identifier of the spawned shell.
+
+ .. attribute:: returncode
+
+ Return code of the process when it exited. A ``None`` value indicates
+ that the process has not terminated yet.
+
+ A negative value ``-N`` indicates that the child was terminated by signal
+ ``N`` (Unix only).
+
+
+Subprocess examples
+===================
+
+Subprocess using transport and protocol
+---------------------------------------
+
+Example of a subprocess protocol using to get the output of a subprocess and to
+wait for the subprocess exit. The subprocess is created by the
+:meth:`BaseEventLoop.subprocess_exec` method::
import asyncio
- import os
import sys
- from asyncio import subprocess
+
+ class DateProtocol(asyncio.SubprocessProtocol):
+ def __init__(self, exit_future):
+ self.exit_future = exit_future
+ self.output = bytearray()
+
+ def pipe_data_received(self, fd, data):
+ self.output.extend(data)
+
+ def process_exited(self):
+ self.exit_future.set_result(True)
@asyncio.coroutine
- def getstatusoutput(*args):
- proc = yield from asyncio.create_subprocess_exec(
- *args,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT)
- try:
- stdout, _ = yield from proc.communicate()
- except:
- proc.kill()
- yield from proc.wait()
- raise
- exitcode = yield from proc.wait()
- return (exitcode, stdout)
+ def get_date(loop):
+ code = 'import datetime; print(datetime.datetime.now())'
+ exit_future = asyncio.Future(loop=loop)
- if os.name == 'nt':
+ # Create the subprocess controlled by the protocol DateProtocol,
+ # redirect the standard output into a pipe
+ create = loop.subprocess_exec(lambda: DateProtocol(exit_future),
+ sys.executable, '-c', code,
+ stdin=None, stderr=None)
+ transport, protocol = yield from create
+
+ # Wait for the subprocess exit using the process_exited() method
+ # of the protocol
+ yield from exit_future
+
+ # Close the stdout pipe
+ transport.close()
+
+ # Read the output which was collected by the pipe_data_received()
+ # method of the protocol
+ data = bytes(protocol.output)
+ return data.decode('ascii').rstrip()
+
+ if sys.platform == "win32":
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
else:
loop = asyncio.get_event_loop()
- coro = getstatusoutput(sys.executable, '-m', 'platform')
- exitcode, stdout = loop.run_until_complete(coro)
- if not exitcode:
- stdout = stdout.decode('ascii').rstrip()
- print("Platform: %s" % stdout)
+
+ date = loop.run_until_complete(get_date(loop))
+ print("Current date: %s" % date)
+ loop.close()
+
+
+Subprocess using streams
+------------------------
+
+Example using the :class:`~asyncio.subprocess.Process` class to control the
+subprocess and the :class:`StreamReader` class to read from the standard
+output. The subprocess is created by the :func:`create_subprocess_exec`
+function::
+
+ import asyncio.subprocess
+ import sys
+
+ @asyncio.coroutine
+ def get_date():
+ code = 'import datetime; print(datetime.datetime.now())'
+
+ # Create the subprocess, redirect the standard output into a pipe
+ create = asyncio.create_subprocess_exec(sys.executable, '-c', code,
+ stdout=asyncio.subprocess.PIPE)
+ proc = yield from create
+
+ # Read one line of output
+ data = yield from proc.stdout.readline()
+ line = data.decode('ascii').rstrip()
+
+ # Wait for the subprocess exit
+ yield from proc.wait()
+ return line
+
+ if sys.platform == "win32":
+ loop = asyncio.ProactorEventLoop()
+ asyncio.set_event_loop(loop)
else:
- print("Python failed with exit code %s:" % exitcode, flush=True)
- sys.stdout.buffer.write(stdout)
- sys.stdout.buffer.flush()
+ loop = asyncio.get_event_loop()
+
+ date = loop.run_until_complete(get_date())
+ print("Current date: %s" % date)
loop.close()
diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -634,7 +634,7 @@
``None``, if no data should be sent to the child. The type of *input*
must be bytes or, if *universal_newlines* was ``True``, a string.
- :meth:`communicate` returns a tuple ``(stdoutdata, stderrdata)``.
+ :meth:`communicate` returns a tuple ``(stdout_data, stderr_data)``.
Note that if you want to send data to the process's stdin, you need to create
the Popen object with ``stdin=PIPE``. Similarly, to get anything other than
--
Repository URL: https://hg.python.org/cpython
More information about the Python-checkins
mailing list