[Python-Dev] Status of packaging in 3.3

[Nick Coghlan]

On Wed, Jun 20, 2012 at 9:31 PM, Tarek Ziadé <tarek at ziade.org> wrote:
> Yeah maybe this subset could be left in 3.3
>
> and we'd remove packaging-the-installer part (pysetup, commands, compilers)
>
> I think it's a good idea !

OK, to turn this into a concrete suggestion based on the packaging docs.

Declare stable, include in 3.3
------------------------------------------
    packaging.version — Version number classes
    packaging.metadata — Metadata handling
    packaging.markers — Environment markers
    packaging.database — Database of installed distributions

Maybe needed as dependencies for above?
------------------------------------------------
    packaging.errors — Packaging exceptions
    packaging.util — Miscellaneous utility functions

It seems to me that stripping the library, docs and tests back to just
these 4 modules and their dependencies shouldn't be much harder than
stripping packaging in its entirety, but my question is what benefit
would we gain from having these (and just these) in the 3.3 stdlib
over having them available on PyPI in distutils2? Third party projects
over the next couple of years are going to want to support more than
just 3.3, so simply depending on distutils2 for this functionality
seems like a far more sensible option.

OTOH, it does send a clear message that progress *is* being made, we
just tried to take too big a jump from implementing these lower level
standards up to "wholesale replacement of distutils" without first
clearly documenting exactly what was wrong with the status quo and
what we wanted to change about it as a sequence of PEPs.

I've broken up the rest of packaging's functionality below into a
series of candidate PEPs that may be more manageable than a single
monolothic "fix packaging" PEP. If we can get multiple interested
parties focusing on different aspects, that may also help with
reducing the burnout factor. Python's current packaging and
distribution story is held together with duct tape and baling wire due
to decisions that were made years ago - unwinding some of those
decisions and replacing them with an alternative that is built on a
solid architectural foundation backed by some level of community
consensus is *not* an easy task, and not one that will be completed
quickly (undue haste will fail the "some level of community consensus"
part, thus missing much of the point of the exercise). That said, I
don't think it's unsolvable either, and there's a reasonable chance to
get something cleaner in place for 3.4.

3.4 PEP: Distutils replacement: Packaging, Distribution & Installation
--------------------------------------------
    # This is one of the big balls of mud w.r.t distutils where third
party projects dig deep into the implementation details because that
is the only way to get things done
    # It may even be the case that this can be broken up even further
    packaging.install — Installation tools
    packaging.dist — The Distribution class
    packaging.manifest — The Manifest class
    packaging.command — Standard Packaging commands
    packaging.command.cmd — Abstract base class for Packaging commands

3.4 PEP: Distutils replacement: Compiling Extension Modules
--------------------------------------------
    # Another big ball of mud. It sounds like the Gentoo folks may
have some feedback in this space.
    packaging.compiler — Compiler classes
    packaging.compiler.ccompiler — CCompiler base class
    packaging.compiler.extension — The Extension class

3.4 PEP: Standard library package downloader (pysetup)
----------------------------------
    # Amongst other things, this needs to have a really good security
story (refusing to install unsigned packages by default, etc)
    packaging.depgraph — Dependency graph builder
    packaging.pypi — Interface to projects indexes
    packaging.pypi.client — High-level query API
    packaging.pypi.base — Base class for index crawlers
    packaging.pypi.dist — Classes representing query results
    packaging.pypi.simple — Crawler using the PyPI “simple” interface
    packaging.pypi.xmlrpc — Crawler using the PyPI XML-RPC interface
    packaging.tests.pypi_server — PyPI mock server
    packaging.fancy_getopt — Wrapper around the getopt module  # Why
does this exist?

3.4 PEP: Simple binary package distribution format
--------------------------------------------------------------------------

    bdist_simple has been discussed enough times, finally seeing a PEP
for it would be nice :)

I think the main lesson to be learned here is that "fix packaging" is
simply too big a task to be managed sensibly. Smaller goals like
"Standardise versioning", "Fix package metadata", "Support uninstall"
are hard enough to achieve, but also provide concrete milestones along
the way to the larger goal.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia