[Distutils] thoughts on distutils 1 & 2

[has]

Hi all,

Sorry for slow reply; was a bit unwell over weekend. Will attempt a 
point-by-point response per topic later if feeling brave <g> (yipe! 
lots of replies). Meantime, here's a more general response to some of 
the areas of discussion/concern that I've picked up on so far:


- Regarding standard package managers on *nix systems (which I've not 
used): does DU intend to wrap these, or plug into them? Current 
impression is it's the former; in which case, what's the reasoning 
for this? I'd have thought it'd make more sense for a system's native 
package manager to "be in charge", as it were, with DU only acting 
under their control. BTW, I've no objection to folks using package 
managers that want to use them; equally, I don't believe folks should 
_have_ to use them (e.g. because runaway complexity or rampant 
lock-in makes any other approach impossible). DU should be able to 
scale up as far as the user wants; but it shouldn't start "high up". 
Nothing's worse than systems that bury themselves (and their users!) 
under the weight of their own complexity.


- Regarding target layout: I realise it's not DU's position to 
_dictate_ layout to systems. What I'm thinking is the packaging 
scheme should be 1. simple, 2. standardised, 3. self-contained, 4. a 
good 'default' layout. Distributing every Python module/extension as 
a Python package (aside from eliminating the confusion over what 
"package" actually means, since the word is currently used to mean 
two different things) would allow DU to invert the current 
distribution format and thereby eliminate a level of folder nesting 
(simpler) and provide a more convenient 'all-in-one' format to kick 
around on a typical system without losing anything (self-contained). 
Folk who are happy with that layout (e.g. would the average 
Mac/Windows user want/need to break the package up?) need do nothing 
more (good 'default' layout). Folk on *nix systems who want to move 
the 'doc' folder to /usr/share/doc/ are free to do so (though copying 
or symlinking it might be better), and, given a standardised layout, 
a generic installer script could easily perform that operation. 
(Ironically, I don't think it's the system's position to _dictate_ 
layout to users either... but arguing with *nix's OCD tendencies is, 
I suppose, ultimately fruitless; it is what it is. [Roll on 
relational file systems...:p])


- Regarding "integration": there are good and bad ways to integrate 
systems. Examples of good: small, distributed, single-task components 
linkable via unix pipes, Apple events, etc. Examples of bad: vast, 
centralised (Soviet bloc-style), "do-it-all" frameworks full of 
wrapper classes round every other system imaginable and more kitchen 
sinks than you can count; inevitably end up riddled with ultra-tight 
coupling, dependencies up the wazoo, and for all the supposed "power" 
never quite manage to do what you want (inflexible). (e.g. See Python 
web framework arena for examples of latter.)


- On splitting various roles of setup.py into individual scripts 
(/src/build.py, /doc/build.py, etc.): aim isn't directly to simplify 
things for developer/user. It's to decouple each functional unit of 
DU, and establish small, simple, open, "generic" interfaces between 
them. This will make each DU component easier for DU developers to 
work on, and easier for DU users to "plug-n-play" with. e.g. DU has a 
good extension building component that could and should stand on its 
own (it may even find uses outside of DU itself), and be easily 
replaceable with alternative build components (based on makefiles, 
scons, etc.); this "ext_build" component would simply become "one of 
the boys" - a peer to all the others, rather than their master (with 
all the additional responsibility and complexity that involves). 
Create "modest", not "Macbeth", code.


- On setup.py providing a very useful indication that "this is a 
DU-installable package": good point; noted. Any system where setup.py 
wasn't ubiquitous would want to provide an equivalent "user-readable 
flag", as it were. (Standardised file+folder structure, presence of 
standard metadata and readme files, etc.)


- Regarding manifests: stupid, brain-dead, error-prone, 
anal-retentive, make-work garbage. These _should_ be eliminated. This 
will do two things: 1. allow a very simple, sensible "default" 
package building behaviour to be instituted (i.e. zip the entire 
folder by default); 2. allow for more intelligent customisation of 
package building behaviour by DU users, who should be able to give 
"smart" instructions like "build package including every item whose 
extension is not in ["pyc", "so"]", instead of having to provide and 
tediously maintain a "dumb" list of filenames.


- On metadata formats: don't really care what format is used as long 
as it is simple, human-readable and -writeable, easily 
machine-parseable, and sufficiently powerful to represent package 
metadata. (e.g. Not sure if standard Mime-like format used for 
PKG-INFO is up to the task: can it do nested structures and lists? 
Format I suggested can do this, and is simple enough that its 
'non-standardness' should not present any problems for adoption.) Oh, 
and a standard meta.txt file in every package means PKG-INFO can also 
be gotten rid of (it's nowt but a weak, clumsy bit of duplicated data 
anyway).


- On PEP262 elimination: again, if users want to create and maintain 
their own database of installed packages using existing package 
manager tools then that's their choice. What's important is that this 
should not be the "default" arrangement, for reasons I've already 
given (synchronisation issues, etc.). By putting metadata into every 
Python package in the filesystem and building an API for retrieving 
that data directly from those packages, you have a solid 
"lowest-common-denominator" foundation that can also meet the 
majority of users' needs for no additional effort. For folk who like 
to maintain a separate DB for efficiency (e.g. on NFS), it also 
provides them the means to easily build that DB given an existing 
installation, not to mention an easy way to rebuild it after it goes 
blooey!


- Basically, goals are: power thru flexibility; flexibility thru 
simplicity; simplicity thru decentralisation. Have simple, sensible, 
effortless defaults (though don't prevent user customising these). 
Decouple different functional areas. Increase genericity. Play nice 
with others (i.e. should cooperate with them, not boss them around). 
Normalise metadata structures and format.


Thoughts?


Regards,

has

p.s. If anyone can point me to good examples of DU-based binary 
distributions, I'd like to take a look to help my understanding, 
thanks.
--
http://freespace.virgin.net/hamish.sanderson/