[Python-Dev] Distutils and Distribute roadmap (and some words on Virtualenv, Pip)

[Toshio Kuratomi]

On Thu, Oct 08, 2009 at 01:27:57PM +0200, M.-A. Lemburg wrote:
> > Tarek Ziadé a écrit :
> >> But if PEP 376 and PEP 386 support are added in Python, we're not far
> >> from being able to provide multiple version support with
> >> the help of importlib.
> 
> Before putting much work into this: do you really think that having
> multiple versions of the same package in the same Python installation
> is a good idea ?
> 
I think it is a good idea.

> Examples:
> What if you have an application that uses two modules which each
> require two different versions of the same package ? Would that
> load the same package twice, causing globals used by these package
> to no work (e.g. isinstance(x, class_name) or global caches) ?
> 
That's not how it should work.  Look at other systems that allow for
installing multiple versions of a library -- for instance, loading dynamic
shared objects in C
* You can install multiple versions of a library in parallel
* The dynamic loader will pick the version of the library that is
  appropriate from the list of available options (the program specifies the
  SONAME it needs -- library name plus API version.  The loader then
  chooses the most recent revision that matches that API version.)
* When one binary needs multiple API versions of the library, the
  application cannot start.

The last point addresses your concern -- depending on multiple, incompatible
versions of a library is prohibited.  The programmer of the application
needs to make the code run with a single version of the code.

> This sounds a lot like DLL- or RPM-hell to me.
> 
RPM-hell (I'm not sure if DLL hell is the same, I have the vague impression
that it is the lack of enough version specification rather than too much but
I don't know for sure). is similar but it takes place on the end-user's
system.  This should take place on the programmer's system instead.
End-users are not in a position to fix things like RPM-hell.  Programmers
are.

Example RPM-hell:
Application Foo requires libbar-1.x
Application Baz requires libbar-2.x

The user may either have Foo or Baz installed on their system with the
appropriate libbar but not both.  They depend on the packagers and
developers of Foo and Bar to do one of the following to resolve the
situation:

* Port Foo and Baz to use the same version of libbar.
* Package libbar in such a way that libbar-1 and libbar-2 are parallel
  installable on the system.  Then they can install two separate packages,
  libbar1-1.0 and libbar2-2.0.

Example of similar Distutils multiple version problem:
The programmer creates an application Foo that depends on python-bar-1.x. He
has recently started work on a file that imports python-baz-1.0.  python-baz
depends on python-bar-2.x.  The first time he tries to run his new code,
python gives him an error message that it is impossible to satisfy the
version requirements for python-bar.  Depending on how the versions are
specified, the error message could be very specific and helpful:

  Impossible version requirements:
    bar Requires: python-baz>=2.0, < 3.0
    foo.py Requires: python-baz >=1.0, < 2.0

The programmer can then discard their new code, port foo.py to
python-baz-2.x, or port python-bar to python-baz-1.x and submit a patch to
the upstream of that module.  Note two things about this scenario:

1) The programmer is the person who is responsible for creating the conflict
and for resolving it.  They are the proper authority for making the decision
to port to python-baz-2.x or not using python-bar.  The end-user who is not
responsible is not impacted by this at all.
2) The programmer would have had to deal with this issue whether we allow
multiple versions to be installed or not.  With multiple version support we
may be able to get them better error messages (depending on how the
dependency information is formatted and how completely it was specified in
the app and modules).

> I think it's much better to keep things simple and under user
> control, e.g. by encouraging use of virtualenv-like setups
> and perhaps adding better native support for these to Python.
> 
> If the user finds that there's a version conflict this can
> then be resolved during environment setup instead of hoping
> for the best and waiting for application failures at run-time.
> 
For the class of user that is actually a developer, it might be somewhat
true that version conflicts should be resolved by them.  But for the class
of user that is an end-user, version conflicts are a totally foreign
concept.  They should be dealt with by the person who is coding the
application for them.

Also note, the ability to have multiple versions makes things easier for
system packagers and provides an easy alternative to a virtualenv for
end-users.

* System packagers: virtualenv does not provide a method suitable for system
  packagers.  The nearest adaptation would be for the system packager to
  install python packages into their own hierarchy not in the PYTHONPATH.
  Then they would need to create a virtualenv-like directory that symlinks
  to the packages in that directory.  Then they would need to write a
  wrapper script for the application that put that virtualenv-like directory
  into the PYTHONPATH before any other site package directories and have
  that wrapper call the real binary.  This is needless complication for the
  typical virtualenv install so the work is not likely to be done there and
  it's incredibly hacky for the system packager so the work is not likely to
  be done there.
* End users: virtualenv creates a whole environment for the application to
  live in.  If python understood how to use multiple versions then we'd only
  need to install the versions of packages that didn't match up with what's
  already on the system into the user's personal site-package directory.
  Then the module loader would take care of loading the module from the
  personal site-packages since it met the version requirements instead of
  the system copy.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20091008/8567bf94/attachment.pgp>