[Pythonmac-SIG] PackMan engine version 0.4

[Bob Ippolito]

On Mar 8, 2004, at 11:01 AM, has wrote:

> Bob wrote:
>
>>> I've been compiling the modules that I've needed; so you could say 
>>> they're the 'right' ones for me. And my environment is straight out 
>>> of the box; I'm not smart enough to modify it, and therefore not 
>>> smart enough to break it either. :)
>>
>> Yes, and that's fine, but many of the modules I offer are ones that 
>> people can't compile on their own without some serious hand holding. 
>> I'd rather just give it to them in binary form than hold everyone's 
>> hand.  I'd rather code than do tech support.. :)
>
> Appreciated, and I have no absolutely problem with binaries being 
> provided for greater convenience. However, it's important that the 
> binary is supplied in addition to a source distribution - not as a 
> substitute for it. Otherwise you're creating lock-in for users. Given 
> the choice between some initial inconvenience in having to build a 
> module myself or not being able to use it at all? I opt for the 
> former. (Recent example: I needed Jack's LaunchServices module for the 
> new version of appscript, but the only version readily available was a 
> 10.3 binary. In the end, I had to track down the source files and 
> write a working setup.py script to install it myself.)
>
> So, at minimum there should always be a source distro available. 
> Additional binaries are optional. Handholding on source distros is 
> also optional: nobody's _obliging_ you to do that. By all means supply 
> the binaries on a strictly unsupported basis. What matters is that  
> the choice belongs to the end user; you shouldn't make it for them.
>
> When I talk about PM's "policy" being flawed, this is partly what I 
> mean. It puts too much responsibility/control in the developer's 
> hands, and not enough in the users'.

I don't supply source distributions because I have only so much time to 
spend on it.  Jack supplies source distributions.  I agree that source 
distributions should be available, but each binary distribution *does* 
have an URL that takes you to an appropriate page for a source 
distribution.  Again, I support 10.3.x users of the vendor supplied 
Python who do not need or want to compile things themselves.  Anyone 
else is not on my radar until the system is more robust.

>>> First, the reason for asking these questions is that I already 
>>> realise PM has been designed to handle the worst-case installations; 
>>> however, in doing so it's completely dropped the ball on 
>>> light-to-average installations.
>>
>> I think you have the wrong impression here.  PM is certainly not 
>> designed to handle worst-case installations.
>
> Worst-case as in complexity, not brittleness.

It's really simple, actually.  All it does is tar zxvf, essentially.. 
except for the source case, where it can execute an arbitrary command 
line.

>> I would say the opposite, it is designed to handle only best-case 
>> situations, and isn't capable of dealing with much deviation without 
>> kicking, screaming, and possible explosion.
>
> Nothing wrong in demanding correctness, as long as achieving that 
> correctness isn't too onerous a task. (e.g. See arguments over strict 
> vs. liberal parsing for the Atom syndication format. If it's 
> unreasonable to expect Atom users to produce well-formed output, then 
> perhaps the problem is with the output format being too damn complex 
> in the first place. In this case, dumping XML for a simple plaintext 
> format would be the obvious answer.)

The XML format is expected to be correct with no deviation.  The 
ambiguity comes from:
	(1) the packages themselves
	(2) the user's environment

Neither of those we have any control over, and it'll be hard to make 
either one bend for us right away.

>>   From what I can tell, the constraint on PM's design was to minimize 
>> development effort (Jack calls this a "straw man" implementation).  
>> Its usability and flexibility sucks because the "engine" is 
>> oversimplified,
>
> Or perhaps the problem it's set for itself is too complex?

I don't really understand this sentence.

>
>> distutils doesn't understand a three level system (vendor/admin/user) 
>> or dyld linking,
>
> Perhaps distutils needs bribed, bullied or bludgeoned into 
> understanding?

Find someone to write that patch, and we can help them understand what 
needs to be done.

>> current versions of Python are linked and installed in strange ways,
>
> Not really my territory; but again, if a way can be found to link and 
> install in non-strange ways, this will be a much more satisfactory 
> solution than jumping through hoops to support the current situation.

10.2.x can not, 10.3.x can.

>> and the GUI implementation is just uh.. awful.
>
> Also redundant. (e.g. PyPI provides a reasonably good graphical 
> interface for browsing/searching for modules; why not try to lever 
> that?)
>
>
>>> Now, this wouldn't be such a problem if PM was positioned as a 
>>> complement to a broader distribution mechanism; the latter taking 
>>> care of light-to-average installations, with PM stepping in only for 
>>> those special-cases that the more general system isn't intended to 
>>> cater for. However, it seems to be marketed as the main [sole?] 
>>> system for _Mac_Python users to install third-party modules.
>>
>> I don't agree with you here.  If PM was better, this wouldn't be an 
>> issue.  It's marketed as the only option to install third-party 
>> modules other than compiling them yourself because well, it *is* the 
>> only option.
>
> This statement is incorrect. The vast majority of third-party modules 
> require no compilation, being written in pure Python. These are 
> precisely the modules PM fails to consider because all its attention 
> is on the small number of modules that do contain C extensions. Worse, 
> there's no way PM can reasonably address this omission, as its 
> reliance on centralised 'gurus' to do all the support work means its 
> scalability is inherently poor.

PackageManager does exactly what it was designed to do.  You want it to 
be something other than what it currently is.

> This is another aspect of PM's flawed "policy", btw; placing too much 
> control/responsibility in the hands of a few individuals upon whom all 
> other developers and users must rely to get anything done. PyPI's 
> decentralised model, where each developer is wholly responsible for 
> managing their own work is much more robust. (BTW, earlier you 
> mentioned the need for platform gurus to ensure third-party modules 
> are MacPython-compatible; the decentralised model in no way prevents 
> this: 'gurus' can work directly with third-party developers to resolve 
> compatibility issues or produce their own repackaged distros as they 
> like.)

That's fine, but the current situation is such that nobody capable and 
willing has the time and effort available to invest in coordinating 
that.  The current situation is such that it requires the least amount 
of coordination possible:  if I have time to make a package available, 
I do.  I'll usually try and send a patch upstream if I feel that it's 
warranted.

>>> So, to summarise where I'm going with this... I see three main areas 
>>> where PM has problems: 1. usability, 2. concept/policy and 3. 
>>> positioning/marketing. The first is what folk are most likely to 
>>> identify problems in simply because it's the most obvious. However, 
>>> although I've found various usability flaws myself, I'm not going to 
>>> ignore these for now and concentrate on what I see as much more 
>>> significant problems in the other two areas.
>>
>> 1. Improvements to the implementation will fix usability.
>
> Not worth spending any time on, however, unless other, deeper issues 
> are resolved first. (i.e. Avoid the temptation to dink around with 
> this just because it's the easiest to resolve.:)

I disagree.  There is a clear problem, people have a hard time 
compiling C extensions, and that is what we have partially solved and 
will more completely solve with a better UI.

>> 2. Could you elaborate on what you mean by concept/policy?
>
> See earlier comments. The whole PM distribution concept is inherently 
> weak, creating a single, central point of failure: your 'platform 
> gurus'. This is something that cannot be fixed through coding.

You're trying to solve a different issue than we are.

>> 3. Because of its "straw man" implementation, it's not even ready for 
>> any real position/marketing.
>
> Then why is it given prominence throughout the MacPython distro? By 
> comparison, distutils - which is far more robust and well-supported - 
> is hardly mentioned. (I'm aware distutils has its faults, but surely 
> it'd be more sensible to fix those directly than invent a new wheel?) 
> Likewise, PyPI is pretty much the standard Python module repository; 
> again, why does MacPython pretty much ignore it when it could lever it 
> to its advantage instead? (PyPI isn't perfect either, of course, but 
> again why not work to improve it?)

We do leverage distutils installation/building mechanisms and PyPI 
metadata in PackageManager's implementation.  Feel free to write up all 
the documentation you want regarding PyPI and distutils on 
pythonmac.org, currently we leave them to do the marketing themselves.  
PackageManager is *NOT* a new wheel, it uses distutils everywhere it 
can (without patching distutils), and all the metadata in the plist 
files is due to PyPI.

> Not really done yet: this is all a bit rough but I'm going to fire it 
> off anyway to keep things moving along. Basically what you want to 
> achieve is the biggest return for the least effort, even if that means 
> [temporarily] compromising a little on the few areas where you 
> currently have the advantage. I've no personal affiliation to 
> distutils or PyPI, but I can spot an obvious winner when its charging 
> right in front of me, and it seems only sensible to hitch one's wagon 
> on for the ride whenever possible.

We already are hitching on distutils and PyPI..

-bob