[Python-Dev] "setuptools has divided the Python community"

[Guido van Rossum]

On Thu, Mar 26, 2009 at 4:33 PM, P.J. Eby <pje at telecommunity.com> wrote:
> At 03:28 PM 3/26/2009 -0500, Guido van Rossum wrote:
>>
>> 2009/3/26 Barry Warsaw <barry at python.org>:
>> > BTW, under a better name, I would support putting pkg_resources in the
>> > stdlib.
>>
>> Last time I looked it was an incredibly complicated piece of code that
>> would have to be refactored considerably before it would be
>> maintainable by the core developers. I never did manage to get a good
>> understanding of the code, but I expect that a lot of the complexity
>> exists so that it works for all Python versions. The stdlib version
>> shouldn't need this -- it should only care about providing a stable
>> API that works with the current version.
>
> As someone else suggested, moving some of the functionality to PEP 302
> interfaces would also help.  Most of the code, though, deals with
> locating/inspecting installed distributions, resolving version requirements,
> and managing sys.path.  And most of the nastiest complexity comes from
> trying to support true filename access to resources -- if that were dropped
> from the stdlib, there'd be no need for egg caches and the like, along with
> all the complexity entailed.
>
> Application environments such as Chandler, Trac, Zope, etc. that want their
> plugins to live in .egg files wouldn't necessarily be able to use such an
> API, but the independent pkg_resources wouldn't be disappearing.  (Of
> course, they could also implement application-specific file extraction, if
> the stdlib API included the ability to inspect and open zipped resources.)
>
> The other significant source of complexity is dynamic management of
> namespace packages; specifically, trying to handle the situation where new
> sys.path entries (e.g. .egg files added as plugins) need to have their
> contents added to existing sys.modules __path__ entries.  This is perhaps
> another feature that could be dropped from the stdlib version, given a way
> to interop with pkg_resources or a replacement.

We had a pretty long discussion about this during the packages session
at the language workshop today at PyCon. Tarek should probably report
more extensively on this (hopefully he won't take it straight to
distutils-sig) but in summary, the plan is roughly:

- standardize more metadata, especially including dependencies, and
APIs for accessing the metadata and dependencies
- there should be an API to get metadata for a package without
actually executing any of the package's installation script
- this will go into the stdlib for Python 2.7 and 3.1
- it will be provided as separately downloadable backports for prior
versions (maybe using a bootstrap not unlike ez_install)
- keep distutils, but start deprecating certain higher-level
functionality in it (e.g. bdist_rpm)
- don't try to provide higher-level functionality in the stdlib, but
instead let third party tools built on top of these core APIs compete

It will be a while, but once this is a reality, it should be easy
enough for the existing crop of tools (like setuptools, pip,
zc.buildout) to be retargeted for the new API, and the benefit would
be that for *consumers* of packages it doesn't matter which tool was
used.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)