[Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement?

Tue Sep 24 02:12:01 CEST 2013

On Sep 23, 2013, at 7:34 PM, Ned Deily <nad at acm.org> wrote:

> In general, I think this is a very important usability feature and I
> am in favor of the general approach.  Good work, all!  I do have some
> comments, primarily about items that are not currently addressed.
> 
>> ``ensurepip`` itself (including the private copy of ``pip`` and its
>> dependencies) will still be installed normally (as it is a regular
>> part of the standard library), only the implicit installation of pip and
>> its dependencies will be skipped.
> 
> The PEP should address the license implications of this since the PEP will
> add the source of pip and setuptools to all new Python releases, both
> source and binary installers releases.  For Python itself, we go to some
> trouble to ensure that all contributions are contributed under a suitable
> license.  What steps are needed, initially and on-going, to ensure that
> these third-party items to be pulled in at each release are compatible
> with the license of Python itself?  Who will do this: release managers?
> Are updates needed in the internal and external documentation for releases
> (README, LICENSE)?  I took a quick look at the current source releases for
> both pip and setuptools.  I noted a reference to the LGPL for pip's CA
> certs bundle; I didn't see a license file for setuptools.

As far as I know the certificate bundle is licensed under either GPL, MPL,
or LGPL. However there is debate about wether a CA bundle *can* be
licensed at all (see https://github.com/pypa/pip/pull/971).

Pip also includes some code taken from other libraries however everything
is licensed as either 1, 2, or 3 clause BSD (besides the aforementioned
certificates). I can't think of us being ok with adding a copyleft license
such as GPL so code we bring in will likely be restricted to things like BSD.

> 
>> This PEP proposes that, if pip still requires it as a dependency,
>> ``ensurepip`` will include a private copy of ``setuptools`` (in addition
>> to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
>> install the private copy in addition to installing ``pip`` itself.
>> 
>> However, this behavior is officially considered an implementation
>> detail. Other projects which explicitly require ``setuptools`` must still
>> provide an appropriate dependency declaration, rather than assuming
>> ``setuptools`` will always be installed alongside ``pip``.
>> 
>> Once pip is able to run ``pip install --upgrade pip`` without needing
>> ``setuptools`` installed first, then the private copy of ``setuptools``
>> will be removed from ``ensurepip`` in subsequent CPython releases.
> 
> The PEP does not sufficiently address the issue of the setuptools-provided
> easy_install command.  I think it is important to do that for several
> reasons:
> 
> 1. The PEP should make clear whether the easy_install scripts are or
> are not installed, globally or in a venv, as a side effect of running
> ensurepip, at least while pip has the implicit private dependency
> on setuptools.  By default, the scripts are installed so ensurepip
> would need to take action to prevent this if that is not desired.
> 
> 2. Despite the current strong preference for using pip as a command
> line installer, the fact is that much third-party documentation still refers
> to and recommends using easy_install for their projects.  That is not
> going to change overnight.  The PEP should identify some steps needed to
> facilitate the migration, like perhaps adding something to the "Installing
> Python Modules" documentation explicitly deprecating easy_install use
> and/or giving pip equivalents for the most common easy_install
> scenarios (or linking to other documentation that does so).
> 
> 3. A specific issue for OS X users is that, as part of OS X, Apple ships
> versions of Python 2 and various third-party packages, including
> setuptools (older, pre-reunification versions, at that) but not pip.
> That means that OS X ships with easy_install and easy_install-2.n
> commands in /usr/bin, including 2.7 in recent OS X releases.  This has
> been an ongoing source of confusion for OS X users who install a newer
> version of Python.  Depending on the distributor and/or source build
> configure options, the new version of Python will be installed to a
> different path, like /usr/local/bin.  Which instance is used
> is generally managed by direct PATH environment manipulations (or by
> launcher programs similar to the Windows py launcher).  If setuptools
> is not installed using the new instance, users invoking the easy_install
> command will by default end up with distributions being installed to a
> system Python rather than to the new Python.  This leads to the
> unfortunately common scenario of:
> 
>    sudo easy_install blah # --> installs to system Python 2.7
>    python -c 'import blah' # --> fails because using newer Python 2.7
> 
> This may very well be the most common current usability problem today
> with python.org (and other third-party) Pythons on OS X.
> 
> With this PEP, we have an opportunity to fix this problem in one of
> two ways.  As long as pip requires setuptools and we do not inhibit
> the installation of the easy_install scripts is to the same directory
> as the python command itself, the new easy_install scripts will
> shadow the Apple-supplied ones just as the python commands do.  If it
> is decided to not install the easy_install scripts now or in the
> future (e.g. when pip no longer depends on setuptools),
> ensurepip could install dummy easy_install scripts (if there
> are none already installed there) that merely emit a message to the user
> to encourage migration to pip and suggest, if needed, to use pip to install
> setuptools to replace the placeholder easy_install.  Either approach
> would be a huge usability improvement over the situation today.
> 
> (The actual implementation details for either approach are slightly more
> complicated for OS X framework builds like the python.org installers
> as the actual scripts directory is a ``bin`` directory within the
> framework itself with optional symlinks from ``/usr/local/bin`` to
> the fw bin directory for specific commands.  The installer will now need
> to also install symlinks in ``/usr/local/bin`` for ``pip`` and, as
> proposed above, for ``easy_install``.)

So this is an interesting question. On one hand I would say we shouldn't be
installing easy_install as that feels very user facing to me and the fact that
setuptools is being installed is an implementation detail. On the other hand
if we put in stub commands that just simply direct the user to use pip then
people who *want* to use easy_install (perhaps for Egg support) won't be
able too (unless perhaps something is released on PyPI that restores the
easy_install commands?)

> 
>> A common source of Python installations are through downstream distributors
>> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
>> package managers [#homebrew]_, or Python-specific tools [#conda]_. In order
> 
> If you are going to call out Homebrew, you should include the other major
> OS X package managers, MacPorts and Fink.
> 
>> .. [#homebrew] `Homebrew  <http://brew.sh/>`
> 
> .. [#macports] `MacPorts  <http://macports.org>`
> .. [#fink] `Fink <http://finkproject.org>`

I can add them, I added homebrew because it's what I use :) Didn't really
think to hard about being super inclusive.

> 
> One final comment is that the PEP does not go into any detail on how it
> will be implemented.  As it stands, there is a fair amount of one-time
> work, including implementing ensurepip, changes to the Windows installer,
> changes to the OS X installer, documentation changes, all with testing and
> backporting to 2.7.x and 3.3.x.  Then there are the on-going process changes
> for all future releases, impacting all release team members, which will need
> to be documented in PEP 101.  Do we have an understanding of who is do the
> big pieces and by when?  For the record, I am willing to do the extra
> one-time and ongoing work for the OS X installers but it would be helpful
> to know how we are going to get it all done in time for upcoming releases.

To further expand upon my original answer, I'm volunteering to do the initial
work on the ensurepip library, the scripts that will automated the ongoing
maintenance work, and the back porting of both of the above. I can also attempt
to do the OSX Installer and Windows installer but I have zero idea how the
installers actually function so it's probably better for someone else to do that.

Since it sounds like you're willing to do the work for the OSX installer that
saves me from having to flail around trying to figure out how to do that part,
so hopefully MvL or someone can do the same with the Windows installer.

I'm not sure what needs done outside of the up front work, do I just propose
changes to PEP 101? Do I make a whole new PEP? Is there more than
just updating PEP 101?

> 
> -- 
> Ned Deily,
> nad at acm.org
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130923/2f69b4a1/attachment-0001.sig>