[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)

[kiorky]

Tarek Ziadé a écrit :
> Here's a quick summary of the main things that are going to happen in
> Distutils, and Distribute, and a few words on virtualenv and pip.
> (there is much much more work going on, but I don't want to drown
> people with details)
> 
> = Distutils =
> 
> Distutils is a package manager and competes with OS package managers.
> This is a good thing because, unless you are developing a library or
> an application that will only run one specific system that has its own
> packaging system like Debian, you will be able to reach much more
> people. Of course the goal is to avoid making the work of a Debian
> packager (or any other OS that has a package manager) too hard. In
> other words, re-packaging a Distutils-based project should be easy and
> Distutils should not get in their way (or as less as possible).
> 
> But right now Distutils is incomplete in many ways and we are trying to fix'em.
> 
> == What's installed ? what's the installation format ? how to uninstall ? ==
> 
> First, it's an incomplete package manager : you can install a
> distribution using it, but there's no way to list installed
> distributions. Worst, you can't uninstall a distribution.
> 
> PEP 376 resolves this, and once it's finished, the goal is to include
> the APIs described there into Distutils itself and into the pkgutil
> module in stdlib. Notice that there's an implementation at
> http://bitbucket.org/tarek/pep376 that is kept up to date with PEP 376
> so people can see what we are talking about.
> 
> Another problem that popped during the last years is the fact that, in
> the same site-packages, depending on the tool that was used to install
> a Distribution, and depending if this distribution uses Distutils or
> Setuptools, you can have different installation formats.
> 
> End-users end up with zipped eggs (one file), unzipped eggs (one
> self-contained format in a directory) and regular Distutils (packages
> and modules in site-packages). And the Metadata are also located in
> many different places depending on the installation format used.
> 
> That can't be. there's no point to keep various installation format in
> the *same* site-packages directory.
> 
> PEP 376 also resolves this by describing a *unique* format that works
> for all. Once this is finished, Distutils will implement it by
> changing the install command accordingly.
> 
> - Work left to do in PEP 376 : restrict its scope to a disk-based,
> file-based site-packages.
> - Goal: 2.7 / 3.2
> 
> == Dependencies ==
> 
> The other feature that makes a packaging system nice is dependencies.
> e.g. a way to list in a distribution, the distributions it requires to
> run. As a matter of fact, PEP 314 has introduced in the Metadata new
> fields for this purpose ("Requires", "Provides and "Obsoletes"). So,
> you can write things like "Requires: lxml >= 2.2.1", meaning that your
> distribution requires lxml 2.2.1 or a newer version to run. But this
> was just description fields and Distutils was not providing any
> feature based on these new fields.
> 
> In fact, no third-party tool either provided a feature based on those
> fields. Setuptools provided "easy_install" a script that looks for the
> dependencies and install them, by querying the Python Package Index
> (PyPI). But this feature was implemented with its own metadata: you
> can add an "install_requires" option in the setup() call in setup.py,
> and it will end up in a "requires.txt" file at installation time that
> is located alongside the Metadata for you distribution.
> 
> So the goal is to review PEP 314 and update the Metadata w.r.t. the
> setuptools feedback and community usage. Once it's done, Distutils
> will implement this new metadata version and promote its usage.
> Promoting its usage means that Distutils will provide some APIs to
> work with these APIs, like a version comparison algorithm.
> 
> And while we're at it, we need to work out some inconsistency with the
> "Author" and "Maintainer" fields. (The latter doesn't exists in the
> Metadata but exists on setup.py side).
> 
> - Work left to do in PEP 314 : finish PEP 386, finish the discussion
> on the "maintainer" field.
> - Goal: 2.7 / 3.2
> 
> == Version comparison ==
> 
> Once you provide dependency fields in the metadata, you need to
> provide a version scheme: a way to compare two versions. Distutils has
> two version comparison algorithms that are not used in its code and in
> only one place in the stdlib where it could be removed with no pain.
> One version scheme is "strict" and one is "loose". And Setuptools has
> another one, which is more heuristic (it will deal with any version
> string and compare it, wether it's wrong or not).
> 
> PEP 386 goal is to describe a version scheme that can be used by all
> and if we can meet a consensus there, we can move on and add
> it as a reference in the update done in  PEP 314, besides the
> dependencies fields. Then, in Distutils we can deprecate the existing
> version
> comparison algorithms and provide a new one based on PEP 386 and
> promote its usage.
> 
> One very important point: we will not force the community to use the
> scheme described in PEP 386, but *there is* already a
> de-facto convention on version schemes at PyPI if you use Pip or
> easy_install, so let's have a documented standard for this,
> and a reference implementation in Distutils.
> 
> There's an implementation at
> http://bitbucket.org/tarek/distutilsversion that is kept up-to-date
> with PEP 386.
> 
> - Work left to do in PEP 386 : another round with the community
> - Goal: 2.7 / 3.2
> 
> == The fate of setup.py, and static metadata ==
> 
> Setup.py is a CLI to create distribution, install them etc. You can
> also use it to retrieve the metadata of a distribution. For
> example you can call "python setup.py --name" and the name will be
> displayed. That's fine. That's great for developers.
> 
> But there's a major flaw: it's Python code. It's a problem because,
> depending on the complexity of this file, an OS packager that just
> wants to get the metadata for the platform he's working on, will run
> arbitrary code that mught do unwanted things (or even that light not
> work)
> 
> So we are going to separate the metadata description from setup.py, in
> a static configuration file, that can be open and read by anyone
> without
> running any code. The only problem with this is the fact that some
> metadata fields might depend on the execution environment. For
> instance, once "Requires" is re-defined and re-introduced via PEP 314,
> we will have cases where "pywin32" will be a dependency to have only
> on win32 systems.
> 
> So we've worked on that lately in Distutils-SIG and came up with a
> micro-language, based on a ConfigParser file, that allows
> writing metadata fields that depends on sys.platform etc. I won't
> detail the syntax here but the idea is that the interpretation
> of this file can be done with a vanilla Python without running arbitrary code.
> 
> In other words : we will be able to get the metadata for a
> distribution without having to install it or to run any setup.py
> command.
> One use case is the ability to list all dependencies a distribution
> requires for a given platform, just by querying PyPI.
> 
> So I am adding this in Distutils for 2.7.
> 
> Of course setup.py stays, and this is backward compatible.
> 
> - Work left to do : publish the final syntax, and do the implementation
> - Goal: 2.7 / 3.2
> 
> == The fate of bdist_* commands ==
> 
> During last Pycon summit we said that we would remove commands like
> bdist_rpm because Python is unable, due to its release cycle,
> to do a good work there. Here's an example: I have from time to time
> cryptic issues in the issue tracker from people from Fedora (or any
> rpm-based system), and I have all the pain in the world for these very
> specific problems to do the proper fix unless some RPM expert helps
> around. And by the time it's detected then fixed, it can be year(s)
> before it's available on their side. That's why, depending on the
> communities, commands like bdist_rpm are just totally ignored, and OS
> packager have their own tools.
> 
> So the best way to handle this is to ask these communities to build
> their own tool and to encourage them to use Distutils as a basis for
> that.
> 
> This does not concern bdist_* commands for win32 because those are
> very stable and don't change too much: Windows doesn't have a package
> manager that would require these commands to evolve with it.
> 
> Anyways, when we said that we would remove bdist_rpm, this was very
> controversial because some people use it and love it.
> 
> So what is going to happen is a status-quo: no bdist_* command will be
> removed but no new bdist_* command wil be added. That's why I've
> encouraged Andrew and Garry, that are working on a bdist_deb command,
> to keep it in the "stdeb" project, and eventually we will
> refer to it in the Distutils documentation if this bdist_deb comply
> with Distutils standard. It doesn't right now because it uses a
> custom version of the Distribution class (through Setuptools) that
> doesn't behave like Distutils' one anymore.
> 
> For Distutils, I'll add some documentation explaining this, and a
> section that will list community-driven commands.
> 
> - Work left to do : write the documentation
> - Goal: 2.7 / 3.2
> 
> = Distribute =
> 
> I won't explain here again why we have forked, I think it's obvious to
> anyone here now. I'll rather explain what
> we are planning in Distribute and how it will interact with Distutils.
> 
> Distribute has two branches:
> 
> - 0.6.x : provides a Setuptools-0.6c9 compatible version
> - 0.7.x : will provide a refactoring
> 
> == 0.6.x ==
> 
> Not "much" is going to happen here, we want this branch to be helpful
> to the community *today* by addressing the 40-or-so bugs
> that were found in Setuptools and never fixed. This is eventually
> happen soon because its development is
> fast : there are up to 5 commiters that are working on it very often
> (and the number grows weekly.)
> 
> The biggest issue with this branch is that it is providing the same
> packages and modules setuptools does, and this
> requires some bootstrapping work where we make sure once Distribute is
> installed, all Distribution that requires Setuptools
> will continue to work. This is done by faking the metadata of
> Setuptools 0.6c9. That's the only way we found to do this.
> 
> There's one major thing though: thanks to the work of Lennart, Alex,
> Martin, this branch supports Python 3,
> which is great to have to speed up Py3 adoption.
> 
> The goal of the 0.6.x is to remove as much bugs as we can, and try if
> possible to remove the patches done
> on Distutils. We will support 0.6.x maintenance for years and we will
> promote its usage everywhere instead of
> Setuptools.
> 
> Some new commands are added there, when they are helpful and don't
> interact with the rest. I am thinking
> about "upload_docs" that let you upload documentation to PyPI. The
> goal is to move it to Distutils
> at some point, if the documentation feature of PyPI stays and starts to be used.
> 
> == 0.7.x ==
> 
> We've started to refactor Distribute with this roadmap in mind (and
> no, as someone said, it's not vaporware,
> we've done a lot already)
> 
> - 0.7.x can be installed and used with 0.6.x
> 
> - easy_install is going to be deprecated ! use Pip !
> 
> - the version system will be deprecated, in favor of the one in Distutils
> 
> - no more Distutils monkey-patch that happens once you use the code
>  (things like 'from distutils import cmd; cmd.Command = CustomCommand')
> 
> - no more custom site.py (that is: if something misses in Python's
> site.py we'll add it there instead of patching it)
> 
> - no more namespaced packages system, if PEP 381 (namespaces package
> support) makes it to 2.7
> 
> - The code is splitted in many packages and might be distributed under
> several distributions.
> 
>    - distribute.resources: that's the old pkg_resources, but
> reorganized in clean, pep-8 modules. This package will
>      only contain the query APIs and will focus on being PEP 376
> compatible. We will promote its usage and see if Pip wants
>      to use it as a basis. And maybe PyPM once it's open source ?
> (<hint> <hint>).
> 	It will probably shrink a lot though, once the stdlib provides PEP 376 support.
> 
>    - distribute.entrypoints: that's the old pkg_resources entry points
> system, but on its own. it uses distribute.resources
> 
>    - distribute.index: that's package_index and a few other things.
> everything required to interact with PyPI. We will promote
>      its usage and see if Pip wants to use it as a basis.
> 
>    - distribute.core (might be renamed to main): that's everything
> else, and uses the other packages.
> 
> 
> Goal: A first release before (or when) Python 2.7 / 3.2 is out.
> 
> 
> = Virtualenv and the multiple version support in Distribute =
> 
> (I am not saying "We" here because this part was not discussed yet
> with everyone)
> 
> Virtualenv allows you to create an isolated environment to install
> some distribution without polluting the
> main site-packages, a bit like a user site-packages.
> 
> My opinion is that this tool exists only because Python doesn't
> support the installation of multiple versions for the same
> distributions.

Not only, for me it exists also to provide a way to isolate/chroot an
application from the host system.


> But if PEP 376 and PEP 386 support are added in Python, we're not far
> from being able to provide multiple version support with
> the help of importlib.
> 
> Setuptools provided a multiple version support but I don't like its
> implementation and the way its works.
> I would like to create a new site-packages format that can contains
> several versions of the same distribution, and :
> 
> - a special import system using importlib that would automatically
> pick the latest version, thanks to PEP 376.
> - an API to force at runtime a specific version (that would be located
> at the beginning of all imports, like __future__)
> - a layout that is compatible with the way OS packagers works with
> python packages
> 
> Goal: a prototype asap (one was started under the "VSP" name (virtual
> site-packages) but not finished yet)
> 
> Regards
> Tarek
> 

-- 
--
Cordialement,
KiOrKY
GPG Key FingerPrint: 0x1A1194B7681112AF


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20091008/61cda4f3/attachment.pgp>