[Python-Dev] Status of packaging in 3.3

[Dag Sverre Seljebotn]

On 06/21/2012 09:05 PM, Tarek Ziadé wrote:
> On 6/21/12 4:26 PM, Dag Sverre Seljebotn wrote:
>>
>>> project should give me so I can compile its extensions ?" I think this
>>> has nothing to do with the tools/implementations.
>> If you sit down and ask your self: "what are the information a python
>>
>> I'm not sure if I understand. A project can't "give the information
>> needed to build it". The build system is an integrated piece of the
>> code and package itself. Making the build of library X work on some
>> ugly HPC setup Y is part of the development of X.
>>
>> To my mind a solution looks something like (and Bento is close to this):
>>
>> Step 1) "Some standard" to do configuration of a package (--prefix and
>> other what-goes-where options, what libraries to link with, what
>> compilers to use...)
>>
>> Step 2) Launch the package's custom build system (may be Unix shell
>> script or makefile in some cases (sometimes portability is not a
>> goal), may be a waf build)
>>
>> Step 3) "Some standard" to be able to cleanly
>> install/uninstall/upgrade the product of step 2)
>>
>> An attempt to do Step 2) in a major way in the packaging framework
>> itself, and have the package just "declare" its C extensions, would
>> not work. It's fine to have a way in the packaging framework that
>> works for trivial cases, but it's impossible to create something that
>> works for every case.
>
> I think we should, as you proposed, list a few projects w/ compilation
> needs -- from the simplest to the more complex, then see how a standard
> *description* could be used by any tool

It's not clear to me what you mean by description. Package metadata, 
install information or description of what/how to build?

I hope you don't mean the latter, that would be insane...it would 
effectively amount to creating a build tool that's both more elegant and 
more powerful than any option that's currently already out there.

Assuming you mean the former, that's what David did to create Bento. 
Reading and understanding Bento and the design decisions going into it 
would be a better use of time than redoing a discussion, and would at 
least be a very good starting point.

But anyway, some project types from simple to advanced:

  - Simple library using Cython + NumPy C API
  - Wrappers around HPC codes like mpi4py, petsc4py
  - NumPy
  - SciPy (uses Fortran compilers too)
  - Library using code generation, Cython, NumPy C API, Fortran 90 code, 
some performance tuning with CPU characteristics (instruction set, cache 
size, optimal loop structure) decided compile-time

>>> And if we're able to write down in a PEP this, e.g. the information a
>>> compiler is looking for to do its job, then any tool out there waf,
>>> scons, cmake, jam, ant, etc, can do the job, no ?
>>>
>>>
>>>>
>>>> Anyway: I really don't want to start a flame-war here. So let's accept
>>>> up front that we likely won't agree here; I just wanted to clarify my
>>>> position.
>>> After 4 years I still don't understand what "we won't agree" means in
>>> this context. *NO ONE* ever ever came and told me : here's what I want a
>>> Python project to describe for its extensions.
>>
>> That's unfortunate. To be honest, it's probably partly because it's
>> easier to say what won't work than come with a constructive
>> suggestion. A lot of people (me included) just use
>> waf/cmake/autotools, and forget about making the code installable
>> through PyPI or any of the standard Python tools. Just because that
>> works *now* for us, but we don't have any good ideas for how to make
>> this into something that works on a wider scale.
>>
>> I think David is one of the few who has really dug into the matter and
>> tried to find something that can both do builds and work through
>> standard install mechanisms. I can't answer for why you haven't been
>> able to understand one another.
>>
>> It may also be an issue with how much one can constructively do on
>> mailing lists. Perhaps the only route forward is to to bring people
>> together in person and walk distutils2 people through some hairy
>> scientific HPC builds (and vice versa).
>
> Like versions scheme, I think it's fine if you guys have a more complex
> system to build software. But there should be a way to share a common
> standard for complation, even if people that uses distutils2 or xxx, are
> just doing the dumbest things, like simple C libs compilation.
>
>>
>>> Just "we won't agree" or "distutils sucks" :)
>>>
>>>
>>> Gosh I hope we will overcome this lock one day, and move forward :D
>>
>> Well, me too.
> The other thing is, the folks in distutils2 and myself, have zero
> knowledge about compilers. That's why we got very frustrated not to see
> people with that knowledge come and help us in this area.

Here's the flip side: If you have zero knowledge about compilers, it's 
going to be almost impossible to have a meaningful discussion about a 
compilation PEP. It's very hard to discuss standards unless everybody 
involved have the necessary prerequisite knowledge. You don't go 
discussing details of the Linux kernel without some solid C experience 
either.

The necessary prerequisites in this case is not merely "knowledge of 
compilers". To avoid repeating mistakes of the past, the prerequisites 
for a meaningful discussion is years of hard-worn experience building 
software in various languages, on different platforms, using different 
build tools.

Look, these problems are really hard to deal with. Myself I have 
experience with building 2-3 languages using 2-3 build tools on 2 
platforms, and I consider myself a complete novice and usually decide to 
trust David's instincts over trying to make up an opinion of my own -- 
simply because I know he's got a lot more experience than I have.

Theoretically it is possible to separate and isolate concerns so that 
one set of people discuss build integration and another set of people 
discuss installation. Problem is that all the problems tangle -- in 
particular when the starting point is distutils!

That's why *sometimes*, not always, design by committee is the wrong 
approach, and one-man-shows is what brings technology forwards.

> So, I reiterate my proposal, and it could also be expressed like this:
>
> 1/ David writes a PEP where he describes how Bento interact with a
> project -- metadata, description files, etc
> 2/ Someone from distutils2 completes the PEP by describing how setup.cfg
> works wrt Extensions
> 3/ we see if we can have a common standard even if it's a subset of
> bento capabilities

bento isn't a build tool, it's a packaging tool, competing directly with 
distutils2. It can deal with simple distutils-like builds using a 
bundled build tool, and currently has integration with waf for 
complicated builds; integration with other build systems will presumably 
be added later as people need it (the main point is that bento is 
designed for it).

Dag