Features for a Python package manager?

[Robert Kern]

Georg Brandl wrote:
> Robert Kern wrote:
> 
>>Mike Meyer wrote:
>>
>>>Nick Coghlan <ncoghlan at iinet.net.au> writes:
>>>
>>>
>>>
>>>>I don't know enough about Portage to answer that question. I do know
>>>>any package manager which made it into the standard distribution would
>>>>need to work for at least the big three platforms (Windows/Mac/*nix) :)
>>>
>>>
>>>Being written in python - and hopefully integrated into Distutils -
>>>why shouldn't it work on any platform that Python worked on?
>>
>>Assumptions about directory structures, and the like.
> 
> 
> That is already taken care by the distutils.

More or less. I'm in the middle of packaging up ~40 Python packages for 
the Mac[1]. For a "standard" packaging mechanism, distutils allows for 
some bloody idiosyncratic ways to say "put these files there". This is a 
hard problem, and it's not solved entirely by distutils.

I don't think anyone has satisfactorily solved the problem of 
distributing data with libraries. Well, okay, the *distribution* isn't 
the problem. Having the library be able to locate that data on all 
platforms is difficult. Some packages will more-or-less hardcode 
*nix-type paths which may be inappropriate even on some *nix-type 
platforms (yes, PyX, I'm looking at you :-)). A general package system 
like Portage has the freedom of being able to dictate these things. A 
Python package manager does not. You can establish a standard for each 
of the Big Three platforms, but it may not do you much good if the 
libraries don't know about it.

CPAN is a closer analogue in this regard and would probably be a better 
tool to study and copy from than Portage. I don't know much about it, 
but how it responds to these issues will probably more instructive than 
how Portage does.

You also have problems with distributing non-Python dependencies like 
libraries. You can sometimes punt and require the user to have already 
installed said libraries and headers to compile against. This will 
probably only work for source distributions on *nixes. You can probably 
get away with appropriately placed DLLs on Windows. Macs may be trickier 
(we Mac users are fussy :-)).

Exercise: VTK with its Python and TCL libraries (why TCL? It's necessary 
to use VTK with Tkinter, which the premier VTK-Python application, 
MayaVi, uses) should package up cleanly for all 3 main platforms. 
External libraries *and* data! It's a joy!

Some other "problem packages"[2] to practice on: PyX, ReportLab, PIL, 
Scipy, matplotlib.

[1] http://www.scipy.org/wikis/featurerequests/MacEnthon
[2] In the sense that they are, to some extent, not painless to package 
up in a very neat way on multiple platforms for one reason or another. 
These are all very fine packages; packaging is a hard problem; and I'm 
grateful for how well they *do* package up so far.

> The various PEPs already
> describe a simple method how to store package metadata, and I will try
> to follow these standards as close as possible. Of course, the PyPI
> would have to be adapted for that.

Fair enough. What parts of Portage do you intend to steal, then?

Please don't read any of this as discouragement. Python package 
management desperately needs a champion, and I'm very glad to see 
someone tackling these issues. I for one fully intend to use as much of 
what you produce as I can to make MacEnthon fully upgradable.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter