[Python-checkins] peps: Update PEP 453 based on MvL's feedback
nick.coghlan
python-checkins at python.org
Thu Sep 19 14:44:52 CEST 2013
http://hg.python.org/peps/rev/538f6c1a90f5
changeset: 5134:538f6c1a90f5
user: Nick Coghlan <ncoghlan at gmail.com>
date: Thu Sep 19 22:43:29 2013 +1000
summary:
Update PEP 453 based on MvL's feedback
- added a new security considerations section
- writing that made it clear it made more sense to just always
install from the private copy, and leave all network access
to the extracted pip
- this design change meant the original module name was no longer
suitable, so it was changed to ``extractpip`` instead
- incorporated Van Lindberg's suggestion from a while back to
rename Tools\Scripts on Windows to bin
files:
pep-0453.txt | 386 ++++++++++++++++++++++----------------
1 files changed, 218 insertions(+), 168 deletions(-)
diff --git a/pep-0453.txt b/pep-0453.txt
--- a/pep-0453.txt
+++ b/pep-0453.txt
@@ -34,15 +34,15 @@
Proposal
========
-This PEP proposes the inclusion of a ``getpip`` bootstrapping module in
+This PEP proposes the inclusion of an ``extractpip`` bootstrapping module in
Python 3.4, as well as in the next maintenance releases of Python 3.3 and
2.7.
-This PEP does *not* propose making pip (or any dependencies) part of the
-standard library. Instead, pip will be a bundled application provided
-along with CPython for the convenience of Python users, but subject to
-its own development life cycle and able to be upgraded independently of
-the core interpreter and standard library.
+This PEP does *not* propose making pip (or any dependencies) directly
+available as part of the standard library. Instead, pip will be a
+bundled application provided along with CPython for the convenience
+of Python users, but subject to its own development life cycle and able
+to be upgraded independently of the core interpreter and standard library.
Rationale
@@ -78,11 +78,11 @@
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
-on Linux systems with platform package managers. No such assistance is
-availabe for Windows and Mac OS X users. The challenges of dealing with
-this problem are a regular feature of feedback the core Python developers
-receive from professional educators and others introducing new users to
-Python.
+on systems with platform package managers (such as major Linux
+distributions). No such assistance is availabe for Windows and
+Mac OS X users. The challenges of dealing with this problem are a
+regular feature of feedback the core Python developers receive from
+professional educators and others introducing new users to Python.
If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
@@ -126,9 +126,10 @@
Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
-on why Python should have a particular tool out of the box instead of
-using the general difficulty of installing third-party packages as
-justification for inclusion.
+on why Python should have a particular tool out of the box, and why it
+is reasonable for that package to adopt the standard library's 18-24 month
+feature release cycle, instead of using the general difficulty of installing
+third-party packages as justification for inclusion.
Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
@@ -163,41 +164,71 @@
Explicit bootstrapping mechanism
================================
-An additional module called ``getpip`` will be added to the standard library
-whose purpose is to install pip and any of its dependencies into the
+An additional module called ``extractpip`` will be added to the standard
+library whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a single
callable named ``bootstrap()`` as well as offer direct execution via
-``python -m getpip``. Options for installing it such as index server,
-installation location (``--user``, ``--root``, etc) will also be available
-to enable different installation schemes.
+``python -m extractpip``.
-It is believed that users will want the most recent versions available to be
-installed so that they can take advantage of the new advances in packaging.
-Since any particular version of Python has a much longer staying power than
-a version of pip in order to satisfy a user's desire to have the most recent
-version the bootstrap will (by default) contact PyPI, find the latest
-version, download it, and then install it. This process is security
-sensitive, difficult to get right, and evolves along with the rest of
-packaging.
+The bootstrap will *not* contact PyPI, but instead rely on a private copy
+of pip stored inside the standard library. Accordingly, only options
+related to the installation location will be supported (``--user``,
+``--root``, etc).
-Instead of attempting to maintain a "mini pip" for the sole purpose of
-installing pip, the ``getpip`` module will, as an implementation detail,
-include a private copy of pip and its dependencies which will be used to
-discover and install pip from PyPI. It is important to stress that this
-private copy of pip is *only* an implementation detail and it should *not*
-be relied on or assumed to exist.
+It is considered desirable that users be strongly encouraged to use the
+latest available version of ``pip``, in order to take advantage of the
+ongoing efforts to improve the security of the PyPI based ecosystem, as
+well as benefiting from the efforts to improve the speed, reliability and
+flexibility of that ecosystem.
-Not all users will have network access to PyPI whenever they run the
-bootstrap. In order to ensure that these users will still be able to
-bootstrap pip the bootstrap will fallback to simply installing the included
-copy of pip. The pip ``--no-download`` command line option will be supported
-to force installation of the bundled version, without even attempting to
-contact PyPI.
+In order to satisfy this goal of providing the most recent version of
+``pip`` by default, the private copy of ``pip`` will be updated in CPython
+maintenance releases, which should align well with the 6-month cycle used
+for new ``pip`` releases.
-This presents a balance between giving users the latest version of pip,
-saving them from needing to immediately upgrade pip after bootstrapping it,
-and allowing the bootstrap to work offline in situations where users might
-already have packages downloaded that they wish to install.
+
+Security considerations
+-----------------------
+
+The design in this PEP has been deliberately chosen to avoid making any
+significant changes to the trust model of the CPython installers for end
+users that do not subsequently make use of ``pip``.
+
+The installers will contain all the components of a fully functioning
+version of Python, including the ``pip`` installer. The installation
+process will *not* require network access, and will *not* rely on
+trusting the security of the network connection established between
+``pip`` and the Python package index.
+
+Only users that choose to use ``pip`` directly will need to pay
+attention to any PyPI related security considerations.
+
+
+Implementation strategy
+-----------------------
+
+To ensure there is no need for network access when installing Python or
+creating virtual environments, the ``extractpip`` module will, as an
+implementation detail, include a complete private copy of pip and its
+dependencies which will be used to extra pip and install it into the target
+environment. It is important to stress that this private copy of pip is
+*only* an implementation detail and it should *not* be relied on or
+assumed to exist beyond the public capabilities exposed through the
+``extractpip`` module (and indirectly through ``venv``).
+
+There is not yet a reference ``extractpip`` implementation. The existing
+``get-pip.py`` bootstrap script demonstrates an earlier variation of the
+general concept, but the standard library version would take advantage of
+the improved distribution capabilities offered by the CPython installers
+to include private copies of ``pip`` and ``setuptools`` as wheel files
+(rather than as embedded base64 encoded data), and would not try to
+contact PyPI (instead installing directly from the private wheel files.
+
+Rather than including separate code to handle the bootstrapping, the
+``extractpip`` module will manipulating sys.path appropriately to allow
+the wheel files to be used to install themselves, either into the current
+Python installation or into a virtual environment (as determined by the
+options passed to the bootstrap command).
Proposed CLI
@@ -207,91 +238,86 @@
options::
Usage:
- python -m getpip [options]
+ python -m extractpip [options]
- Download Options:
- --no-download Install the bundled version, don't attempt to download
- -i, --index-url <url> Base URL of Python Package Index (default https://pypi.python.org/simple/).
- --proxy <proxy> Specify a proxy in the form [user:passwd@]proxy.server:port.
- --timeout <sec> Set the socket timeout (default 15 seconds).
- --cert <path> Path to alternate CA bundle.
+ General Options:
+ -h, --help Show help.
+ -v, --verbose Give more output. Option is additive, and can be used up to 3 times.
+ -V, --version Show the pip version that would be extracted and exit.
+ -q, --quiet Give less output.
Installation Options:
- -U, --upgrade Upgrade pip and dependencies, even if already installed
- --user Install using the user scheme.
- --root <dir> Install everything relative to this alternate root directory.
+ -U, --upgrade Upgrade pip and dependencies, even if already installed
+ --user Install using the user scheme.
+ --root <dir> Install everything relative to this alternate root directory.
-Additional options (such as verbosity and logging options) may also
-be supported.
+In most cases, end users won't need to use this CLI directly, as ``pip``
+should have been installed automatically when installing Python or when
+creating a virtual environment.
+
+Users that want to retrieve the latest version from PyPI, or otherwise
+needing more flexibility, should invoke the extracted ``pip``
+appropriately.
Proposed module API
-------------------
-The proposed ``getpip`` module API is a single ``bootstrap`` function with
-parameter names derived directly from the proposed CLI::
+The proposed ``extractpip`` module API consists of the following two
+functions::
- def bootstrap(download=True, upgrade=False, root=None, user=False,
- index_url=None, cert=None, proxy=None, timeout=15):
+ def version():
+ """Returns a string specifying the bootstrapped version of pip"""
+
+ def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
"""Bootstrap pip into the current Python installation (or the given
root directory)"""
-The only changes are to replace the ``--no-download`` opt-out option with
-the True-by-default ``download`` option and to replace the hyphen in
-``index-url`` with an underscore to create a legal Python identifier.
-
Invocation from the CPython installers
--------------------------------------
-The CPython Windows and Mac OS X installers will each gain two new options:
+The CPython Windows and Mac OS X installers will each gain a new option:
* Install pip (the default Python package management utility)?
-* Upgrade pip to the latest version (requires network access)?
-Both options will be checked by default, with the option to upgrade pip
-being available for selection only if the option to install pip is checked.
+This option will be checked by default.
-If both options are checked, then the installer will invoke the following
+If the option is checked, then the installer will invoke the following
command with the just installed Python::
- python -m getpip --upgrade
-
-If only the "Install pip" option is checked, then the following command will
-be invoked::
-
- python -m getpip --upgrade --no-download
+ python -m extractpip --upgrade
This ensures that, by default, installing or updating CPython will ensure
-that either the latest available version of PyPI is installed (directly from
-PyPI if permitted, otherwise whichever is more recent out of an already
-installed version and the private copy inside ``getpip``)
+that the installed version of pip is at least as recent as the one included
+with that version of CPython.
Installing from source
----------------------
While the prebuilt binary installers will be updated to run
-``python -m getpip`` by default, no such change will be made to the
-``make install`` and ``make altinstall`` commands of the source distribution.
+``python -m extractpip`` by default, no such change will be made to the
+``make install`` and ``make altinstall`` commands of the source
+distribution.
-``getpip`` itself will still be installed normally (as it is a regular
+``extractpip`` itself (including the private copy of ``pip`` and its
+dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.
Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
-``make`` for the ``getpip`` invocation also ensures those installing from
-a custom source build can easily force an offline installation of pip,
-install it from a private index server, or skip installing pip entirely.
+``make`` for the ``extractpip`` invocation avoids any challenges
+associated with determining where to install the extracted ``pip``.
Changes to virtual environments
-------------------------------
Python 3.3 included a standard library approach to virtual Python environments
-through the ``venv`` module. Since it's release it has become clear that very
+through the ``venv`` module. Since its release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
@@ -302,17 +328,22 @@
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
-suitable replacement. To handle cases where a user does not wish to have pip
-bootstrapped into their virtual environment a ``--without-pip`` option will be
-added. The ``--no-download`` option will also be supported, to force the
-use of the bundled ``pip`` rather than retrieving the latest version from
-PyPI.
+suitable replacement.
+
+To handle cases where a user does not wish to have pip bootstrapped into
+their virtual environment a ``--without-pip`` option will be
+added.
The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
-two new parameters: ``with_pip`` (defaulting to ``False``) and
-``bootstrap_options`` (accepting a dictionary of keyword arguments to
-pass to ``getpip.bootstrap`` if ``with_pip`` is set, defaulting to
-``None``).
+one new parameter: ``with_pip`` (defaulting to ``False``).
+
+The new default for the module API is chosen for backwards compatibility
+with the current behaviour (as it is assumed that most invocation of the
+``venv`` module happens through third part tools that likely will not
+want ``pip`` installed without explicitly requesting it), while the
+default for the command line interface is chosen to try to ensure ``pip``
+is available in most virtual environments without additional action on the
+part of the end user.
This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
@@ -335,13 +366,14 @@
Bundling CA certificates with CPython
-------------------------------------
-The reference ``getpip`` implementation includes the ``pip`` CA
-bundle along with the rest of pip. This means CPython effectively includes
-a CA bundle that is used solely for ``getpip``.
+The ``extractpip`` implementation will include the ``pip`` CA bundle along
+with the rest of ``pip``. This means CPython effectively includes
+a CA bundle that is used solely by ``pip`` after it has been extracted.
-This is considered desirable, as it ensures that ``pip`` will behave the
-same across all supported versions of Python, even those prior to Python
-3.4 that cannot access the system certificate store on Windows.
+This is considered preferable to relying solely on the system
+certificate stores, as it ensures that ``pip`` will behave the same
+across all supported versions of Python, even those prior to Python 3.4
+that cannot access the system certificate store on Windows.
Automatic installation of setuptools
@@ -354,11 +386,9 @@
when Python 3.4.0 is released).
This PEP proposes that, if pip still requires it as a dependency,
-``getpip`` will include a private copy of ``setuptools`` (in addition
-to the private copy of ``pip``). In normal operation, ``python -m getpip``
-will then download and install the latest version of ``setuptools`` from
-PyPI (as a dependency of ``pip``), while ``python -m getpip --no-download``
-will install the private copy.
+``extractpip`` will include a private copy of ``setuptools`` (in addition
+to the private copy of ``extractpip``). ``python -m extractpip`` will then
+install the private copy in addition to installing ``pip`` itself.
However, this behaviour is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
@@ -367,52 +397,32 @@
Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
-will be removed from ``getpip``.
+will be removed from ``extractpip`` in subsequent CPython releases.
-Updating the bundled pip
-------------------------
+Updating the private copy of pip
+--------------------------------
In order to keep up with evolutions in packaging as well as providing users
-who are using the offline installation method with as recent version a
-possible the ``getpip`` module will be regularly updated to the latest
-versions of everything it bootstraps.
+with as recent version a possible the ``extractpip`` module will be
+regularly updated to the latest versions of everything it bootstraps.
-After each new pip release, and again during the preparation for any
+After each new ``pip`` release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of this PEP, will be run to ensure the private copies stored in the CPython
source repository have been updated to the latest versions.
-Updating the getpip module API and CLI
---------------------------------------
+Updating the extractpip module API and CLI
+------------------------------------------
-Future security updates for pip and PyPI (for example, automatic
-verification of package signatures) may also provide desirable security
-enhancements for the ``getpip`` bootstrapping mechanism.
+Like ``venv`` and ``pyvenv``, the ``extractpip`` module API and CLI
+will be governed by the normal rules for the standard library: no
+new features are permitted in maintenance releases.
-It is desirable that these features be made available in standard library
-maintenance releases, not just new feature releases.
-
-Accordingly, a slight relaxation of the usual "no new features in
-maintenance releases" rule is proposed for the ``getpip`` module. This
-relaxation also indirectly affects the new ``bootstrap_options`` parameter
-in the ``venv`` module APIs.
-
-Specifically, new security related flags will be permitted, with the
-following restrictions:
-
-- for compatibility with third-party usage of ``getpip`` module (for
- example, with a private index server), any such flag must be *off* by
- default in maintenance releases. It *should* be switched on by
- default in the next feature release.
-- the CPython installers and the ``pyvenv`` CLI in the affected maintenance
- release should explicitly opt-in to the enhanced security features when
- automatically bootstrapping ``pip``
-
-This means that maintenance releases of the CPython installers will
-benefit from security enhancements by default, while avoiding breaking
-customised usage of the bootstrap mechanism.
+However, the embedded components may be updated as noted above, so
+the extracted ``pip`` may offer additional functionality in maintenance
+releases.
Feature addition in maintenance releases
@@ -433,12 +443,6 @@
than only those with the freedom to adopt Python 3.4 as soon as it is
released.
-This is also a matter of starting as we mean to continue: as noted above,
-``getpip`` will have a limited permanent exemption from the "no new
-features in maintenance releases" restriction, as it will include (and
-rely on) upgraded private copies of ``pip`` and ``setuptools`` even in
-maintenance releases, and may offer new security related options itself.
-
Open Question: Uninstallation
=============================
@@ -463,21 +467,44 @@
Open Question: Script Execution on Windows
==========================================
+.. note::
+
+ Perhaps this question should be separated out from the PEP?
+
While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
-include the Scripts directory. This PEP proposes that this installer option
-be changed to also add the Scripts directory to PATH (either always, or
-else as a checked by default suboption).
+include the Scripts directory. Independently of this PEP, a proposal has
+also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in
+order to improve consistency with the typical script installation directory
+names on \*nix systems.
-Without this change, the most reliable way to invoke pip on Windows (without
-tinkering manually with PATH) is actually ``py -m pip`` (or ``py -3 -m pip``
-to select the Python 3 version if both Python 2 and 3 are installed)
-rather than simply calling ``pip``.
+Accordingly, in addition to adding the option to extract and install ``pip``
+during installation, this PEP proposes that the Windows installer (and
+``sysconfig``) in Python 3.4 and later be updated to:
+
+- install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts
+- add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the
+ PATH modification option is enabled during installation
+
+For Python 2.7 and 3.3, it is proposed that the only change be the one
+to bootstrap ``pip`` by default.
+
+This means that, for Python 3.3, the most reliable way to invoke pip on
+Windows (without tinkering manually with PATH) will actually be
+``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both
+Python 2 and 3 are installed) rather than simply calling ``pip``.
+
+For Python 2.7 and 3.2, the most reliable mechanism will be to install the
+standalone Python launcher for Windows and then use ``py -m pip`` as noted
+above.
Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip`` needed only to select a non-default version in
-the parallel installation case (and outside a virtual environment).
+the parallel installation case (and outside a virtual environment). This
+change should also make the ``pyvenv`` command substantially easier to
+invoke on Windows, along with all scripts installed by ``pip``,
+``easy_install`` and similar tools.
While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
@@ -510,9 +537,9 @@
downstream distributors have already made exception to the common
"debundling" policy.
* This does mean that if ``pip`` needs to be updated due to a security
- issue, so does the bundled version in the ``getpip`` bootstrap module
+ issue, so does the bundled version in the ``extractpip`` bootstrap module
* However, altering the bundled version of pip to remove the embedded
- CA certificate bundle and rely the system CA bundle instead is a
+ CA certificate bundle and rely on the system CA bundle instead is a
reasonable change.
* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
@@ -522,14 +549,17 @@
new metadata formats which may not have a ``setup.py``.
* Ensure that all features of this PEP continue to work with any modifications
- made.
+ made to the redistributed version of Python.
- * Online installation of the latest version of pip into a global or virtual
- python environment using ``python -m getpip``.
- * Offline installation of the bundled version of pip into a global or virtual
- python environment using ``python -m getpip``.
+ * Checking the version of pip that will be bootstrapped using
+ ``python -m extractpip --version`` or ``extractpip.version()``.
+ * Installation of the bundled version of pip into a global or virtual
+ python environment using ``python -m extractpip`` or
+ ``extractpip.bootstrap()``.
* ``pip install --upgrade pip`` in a global installation should not affect
- any already created virtual environments.
+ any already created virtual environments (but is permitted to affect
+ future virtual environments, even though it will not do so in the
+ normal case).
* ``pip install --upgrade pip`` in a virtual environment should not affect
the global installation.
@@ -560,10 +590,9 @@
Backwards Compatibility
-----------------------
-Except for security enhancements (as noted above), the public API of the
-``getpip`` module itself will fall under the typical backwards compatibility
-policy of Python for its standard library. The externally developed software
-that this PEP bundles does not.
+The public API and CLI of the ``extractpip`` module itself will fall under
+the typical backwards compatibility policy of Python for its standard
+library. The externally developed software that this PEP bundles does not.
Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
@@ -573,16 +602,37 @@
Security Releases
-----------------
-Any security update that affects the ``getpip`` module will be shared prior to
-release with the Python Security Response Team (security at python.org). The
-PSRT will then decide if the reported issue warrants a security release of
-CPython.
+Any security update that affects the ``extractpip`` module will be shared
+prior to release with the Python Security Response Team
+(security at python.org). The PSRT will then decide if the reported issue
+warrants a security release of CPython with an updated private copy of
+``pip``.
Appendix: Rejected Proposals
============================
+Automatically contacting PyPI when bootstrapping pip
+----------------------------------------------------
+
+Earlier versions of this PEP called the bootstrapping module ``getpip`` and
+defaulted to downloading and installing ``pip`` from PyPI, with the private
+copy used only as a fallback option or when explicitly requested.
+
+This resulted in several complex edge cases, along with difficulties in
+defining a clean API and CLI for the bootstrap module. It also significantly
+altered the default trust model for the binary installers published on
+python.org, as end users would need to explicitly *opt-out* of trusting
+the security of the PyPI ecosystem (rather than opting in to it by
+explicitly invoking ``pip`` following installation).
+
+As a result, the PEP was simplified to the current design, where the
+bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI
+is now always an explicit seperate step, with direct access to the full
+pip interface.
+
+
Implicit bootstrap
------------------
--
Repository URL: http://hg.python.org/peps
More information about the Python-checkins
mailing list