[Distutils] Library instability on PyPI and impact on OpenStack

[Mark McLoughlin]

On Mon, 2013-03-04 at 12:44 -0500, Donald Stufft wrote:
> On Monday, March 4, 2013 at 12:36 PM, Mark McLoughlin wrote:

> > If parallel incompatible installs is a hopeless problem in Python,
> > why
> > the push to semantic versioning then rather than saying that
> > incompatible API changes should mean a name change?
> Forcing a name change feels ugly as all hell. I don't really see what
> parallel installs has much to do with anything. I don't bundle anything
> and i'm ideologically opposed to it generally but I don't typically have
> a need for parallel installs because I use virtual environments. Why
> don't you utilize those? (Not being snarky, actually curious).

It's a fair question.

To answer it with a question, how do you imagine Linux distributions
using virtual environments such that:

  $> yum install -y openstack-nova

uses a virtual environment? How does it differ from bundling? (Not being
snarky, actually curious :)

The approach that some Fedora folks are trying out is called "Software
Collections". It's not Python specific, but it's basically the same as a
virtual environment.

For OpenStack, I think we'd probably have all the Python libraries we
require installed under e.g. /opt/rh/openstack-$version so that you
could have programs from two different releases of OpenStack installed
on the same system.

Long time packagers are usually horrified at this idea e.g.

  http://lists.fedoraproject.org/pipermail/devel/2012-December/thread.html#174872

Some of the things to think about:

  - Each of the Python libraries under /opt/rh/openstack-$version would 
    come from new packages like openstack-$version-python-eventlet.rpm -
    how many applications in Fedora would have a big stack of "bundled" 
    python packages like OpenStack? 5, 10, 50, 100? Let's say it's 10 
    and each one stack has 20 packages. That's 200 new packages which 
    need to be maintained by Fedora versus the current situation where 
    we (painfully) make a single stack of libraries work for all 
    applications.

  - How many of these 200 new packages are essentially duplicates? Once 
    you go down the route of having applications bundle libraries like 
    this, there's going to basically be no sharing.

  - What's the chance that that all of these 200 packages will be kept 
    up to date? If an application works with a given version of a 
    library and it can stick with that version, it will. As a Python 
    library maintainer, wow do you like the idea of 10 different
    versions of you library included in Fedora?

  - The next time a security issue is found in a common Python library,
    does Fedora now have to rush out 10 parallel fixes for it?

You can see that reaction in mails like this:

  http://lists.fedoraproject.org/pipermail/devel/2012-December/174944.html

and the "why can't these losers just maintain compatibility" view:

  http://lists.fedoraproject.org/pipermail/devel/2012-December/175028.html
  http://lists.fedoraproject.org/pipermail/devel/2012-December/174929.html

Notice folks complaining about Ruby and Java here, not Python. I can see
Python embracing semantic versioning and "just use venv" shortly leading
to Python being included in the list of "heretics".

Thanks,
Mark.

[1] - http://docs.fedoraproject.org/en-US/Fedora_Contributor_Documentation/1/html-single/Software_Collections_Guide/index.html