[Python-Dev] Status of packaging in 3.3

[Dag Sverre Seljebotn]

On 06/21/2012 11:04 PM, Tarek Ziadé wrote:
> On 6/21/12 10:46 PM, Dag Sverre Seljebotn wrote:
> ...
>>> I think we should, as you proposed, list a few projects w/ compilation
>>> needs -- from the simplest to the more complex, then see how a standard
>>> *description* could be used by any tool
>>
>> It's not clear to me what you mean by description. Package metadata,
>> install information or description of what/how to build?
>>
>> I hope you don't mean the latter, that would be insane...it would
>> effectively amount to creating a build tool that's both more elegant
>> and more powerful than any option that's currently already out there.
>>
>> Assuming you mean the former, that's what David did to create Bento.
>> Reading and understanding Bento and the design decisions going into it
>> would be a better use of time than redoing a discussion, and would at
>> least be a very good starting point.
>
> What I mean is : what would it take to use Bento (or another tool) as
> the compiler in a distutils-based project, without having to change the
> distutils metadata.

As for current distutils/setuptools/distribute metadata, the idea is you 
run the bento conversion utility to convert it to Bento metadata, then 
use Bento.

Please read

http://cournape.github.com/Bento/

There may be packages where this doesn't work and you'd need to tweak 
the results yourself though.

>> Here's the flip side: If you have zero knowledge about compilers, it's
>> going to be almost impossible to have a meaningful discussion about a
>> compilation PEP. It's very hard to discuss standards unless everybody
>> involved have the necessary prerequisite knowledge. You don't go
>> discussing details of the Linux kernel without some solid C experience
>> either.
> Consider me as the end user that want to have his 2 C modules compiled
> in their Python project.

OK, so can I propose that you kill off distutils2 and use bento 
wholesale instead?

Obviously not. So you're not just an end-user. That illusion would wear 
rather thin very quickly.

>>
>> The necessary prerequisites in this case is not merely "knowledge of
>> compilers". To avoid repeating mistakes of the past, the prerequisites
>> for a meaningful discussion is years of hard-worn experience building
>> software in various languages, on different platforms, using different
>> build tools.
>>
>> Look, these problems are really hard to deal with. Myself I have
>> experience with building 2-3 languages using 2-3 build tools on 2
>> platforms, and I consider myself a complete novice and usually decide
>> to trust David's instincts over trying to make up an opinion of my own
>> -- simply because I know he's got a lot more experience than I have.
>>
>> Theoretically it is possible to separate and isolate concerns so that
>> one set of people discuss build integration and another set of people
>> discuss installation. Problem is that all the problems tangle -- in
>> particular when the starting point is distutils!
>>
>> That's why *sometimes*, not always, design by committee is the wrong
>> approach, and one-man-shows is what brings technology forwards.
>
> I am not saying this should be designed by a commitee, but rather - if
> such a tool can be made compatible with simple Distutils project, the
> guy behind this tool can probably help on a PEP with feedback from a
> larger audience than the sci community.
>
> What bugs me is to say that we live in two separate worlds and cannot
> build common pieces. This is not True.

I'm not saying it's *impossible* to build common pieces, I'm suggesting 
that it's not cost-effective in terms of man-hours going into it. And 
the problem isn't technical as much as social and the mix of people and 
skill sets involved.

But David really made that decision for me when he left distutils-sig, 
I'm not going to spend my own time and energy trying to get decent 
builds shoehorned into distutils2 when he is busy working on a solution.

(David already spent loads of time on trying to integrate scons with 
distutils (the numscons project) and maintained numpy.distutils and 
scipy builds for years; I trust his judgement above pretty much anybody 
else's.)

>>> So, I reiterate my proposal, and it could also be expressed like this:
>>>
>>> 1/ David writes a PEP where he describes how Bento interact with a
>>> project -- metadata, description files, etc
>>> 2/ Someone from distutils2 completes the PEP by describing how setup.cfg
>>> works wrt Extensions
>>> 3/ we see if we can have a common standard even if it's a subset of
>>> bento capabilities
>>
>> bento isn't a build tool, it's a packaging tool, competing directly
>> with distutils2. It can deal with simple distutils-like builds using a
>> bundled build tool, and currently has integration with waf for
>> complicated builds; integration with other build systems will
>> presumably be added later as people need it (the main point is that
>> bento is designed for it).
> I am not interested in Bento-the-tool. I am interested in what such a
> tool needs from a project to use it =>

Again, you should read the elevator pitch at 
http://cournape.github.com/Bento/ + the Bento documentation.

> "It can deal with simple distutils-like builds using a bundled build
> tool" => If I understand this correctly, does that mean that Bento can
> build a distutils project with the distutils Metadata ?

Sorry, what I meant with "distutils-like builds" is "two simple C 
extensions", i.e. the trivial build case.

Dag