[Distutils] "Python Package Management Sucks"

[Chris Withers]

Matthias Klose wrote:
>>> Install debian and get back to productive tasks.
>> This is an almost troll-like answer.
>> See page 35 of the presentation.
> 
> I disagree. You could think of "Packages are Pythons Plugins" (taken
> from page 35) as a troll-like statement as well.

You're welcome to your (incorrect) opinion ;-)
Debian packages could just as easilly be seen as Debian's pluggins.

> and a subset of all these operating systems (linux or other) do have
> the need to distribute python and a set of python modules and
> extensions. they cannot rely on a plugin system outside the (os)
> distribution.

OK, you guys have persuaded me of this at least...

>> - all the package management systems behave differently and expect 
>> packages to be set up differently for them
> 
> correct, but again they share common requirements.

...but all have different implementations.

> some people prefer to name this "stable releases" instead of
> "bitrot". 

I'll call bullshit on this one. The most common problem I have as a 
happy Debian user and advocate when I go to try and get help for a 
packaged application (I use packages because I perhaps mistakenly assume 
this is the best way to get security-fixed softare), such as postfix, 
postgres, and Zope if I was foolish enough to take that path, is "why 
are toy using that ancient and buggy version of the software?!" shortly 
before pointing out how all the issues I'm facing are solved in newer 
(stable) releases.

The problem is that first the application needs to be tested and 
released by its community, then Debian needs to re-package, patch, 
generally mess around with it, etc before it eventually gets a "Debian 
release". It's bad enough with apps with huge support bases like 
portgres, imagine trying to do this "properly" for the 4000-odd packages 
on PyPI...

> Speaking of extensions "maintained by the entity originating the
> python package": this much too often is a way of bitrot. is the
> shipped library up to date? does it have security fixes? how many
> duplicates are shipped in different extensions? does the library need
> to be shipped at all (because some os does ship it)?

So what do you propose doing one projectA depends on version 1.0 of libC 
and projectB depends on version 2.0 of libC?

> this is known trouble for os distributors, and your statement is
> generally wrong. firefox plugins are packaged in distributions and the
> plugin system is able to cope with packaged plugins.

I guess since my desktop OS is still windows, this is not something I've 
had to fight with ;-)

>> Packages are Python's "plugins" and so should get the same type of
>> consistent, cross-platform package management targetted at the
>> application in question, which is Python in this case.
> 
> No, as explained above. 

While I'll buy the argument that python packaging tools should make life 
easier for production of os-specific packages, I still don't think 
you're correct ;-)

> Considering an extension interfacing a library
> shipped with the os, you do want to use this library, not add another
> copy. 

libxml2 seems to be agood example to use here...

I guess on debian I'd need to likely install libxml2-dev before I could 
install the lxml package...

...what about MacOS X?

...what about Windows?

> An upstream
> extension maintainer cannot provide this unless he builds this
> extension for every (os) distribution and maintains it during the os'
> lifecycle.

...or just says in the docs "hey, you need libxml2 for this, unless 
you're on Windows, in which case the binary includes it".

>  - os distributors usually try to minimize the versions they include,
>    trying to just ship one version.  

...which is fair enough for the "system python", but many of us have a 
collection of apps, some of which require Python 2.4, some Python 2.5, 
and on top of each of those, different versions of different packages 
for each app.

In my case, I do source (alt-)installs of python rather than trusting 
the broken stuff that ships with Debian and buildout to make sure I get 
the right versions of the right packages for each project.

>  - setuptools has the narrow minded view of a python package being
>    contained in a single directory, which doesn't fit well when you
>    do have common locations for include or doc files. 

Python packages have no idea of "docs" or "includes", which is certainly 
a deficiency.

> way packaging the python module with rpm or dpkg. E.g. namespace
> packages are a consequence how setuptools distributes and installs
> things. Why force this on everybody?

being able to break a large lump (say zope.*) into seperate 
distributions is a good idea, which setuptools implements very badly 
using namespace packages...

> A big win could be a modularized setuptools where you are able to only
> use the things you do want to use, e.g.
> 
>  - version specifications (not just the heuristics shipped with
>    setuptools).

not sure what you mean by this.

>  - specification of dependencies.
> 
>  - resource management

?

>  - a module system independent from any distribution specific stuff.

?

>  - any other distribution specific stuff.

?

> The "Wouldn't it be nice if?" pacman (page 55) sounds like a nice
> idea, if it could just handle multiple repositories and one of them
> being the archive of the os distributor (iirc the java jsr277 proposed
> the use of multiple repositories, even in different formats).

Where does it say it doesn't support multiple repositories? ;-)

Chris

-- 
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk