[Distutils] waf can be used to compile Python extension modules

[David Cournapeau]

On Fri, Mar 21, 2014 at 1:46 PM, Daniel Holth <dholth at gmail.com> wrote:

> Bento emulates setup.py:
> https://github.com/cournape/Bento/blob/master/setup.py
>
> It mostly has to implement egg_info and install to be pip compatible.
>

Yes. While some things are quite not ideal in Bento, I think that part is a
good one: build a system that goes away completely from distutils, and then
emulate those interfaces (where it makes sense).

This allows compatibility in most cases, while avoiding re-introducing what
makes distutils horrible (commands and compilers).

Before attempting to solve the build problem itself, I would suggest to
have a small library extracted from distutils that does not impose any
'policy' on the build, but expose compilation information (flags, link
options, etc...). This library could be used from distutils, but also
(crucially) by 3rd party tools.

Regarding build, my own experience after having dealt with unholly
scons/waf/distutils/numpy.distutils integration projects:

  - solving the simple cases (simple C extensions) is easy, but moving to
more complex cases makes the problem much harder. Build is really a case
where solving 80 % of the usecases is solving 10 % of the problem. Solving
80 % of the usecases is ok as long as it does not prevent the remaining 20
% to do what they want.
  - neither scons or waf are really usable as libraries (scons much more so
than waf). Even fbuild, which I generally find more approachable, has a
'god' object that makes integration difficult.
  - having well documented, self-contained API to build final artefacts
(sdist, installs, eggs, wheels, bdist_wininst, bdist_msi, etc...) that help
integrating with 3rd party build tools is more critical than the build tool
itself.

David

>
> In the future we'd like them to implement dist_info (generate a
> .dist-info directory instead of an .egg-info directory) and wheel; an
> sdist would signal that it doesn't implement setup.py install and pip
> would always build and then install a wheel. This would be a
> relatively simple thing to add to pip.
>
> Once setup.py emulation is working then we could define a Python-level
> plugin interface for build systems that might return metadata dicts,
> and also provide a setup.py that talks to this plugin interface. A
> later version of pip might talk directly to the plugin without
> bothering with setup.py.
>
> This strategy does not generally try to eliminate arbitrary code
> execution during builds - builds are an inherently arbitrary-code
> process. But once the build has happened most installs should work
> without arbitrary code execution.
>
> On Fri, Mar 21, 2014 at 6:13 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > On 20 Mar 2014 23:16, "Brett Cannon" <bcannon at gmail.com> wrote:
> >>
> >>
> >>
> >> On Thu Mar 20 2014 at 12:51:13 AM, Nick Coghlan <ncoghlan at gmail.com>
> >> wrote:
> >>>
> >>> On 20 March 2014 09:54, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> >>> > Daniel Holth <dholth <at> gmail.com> writes:
> >>> >
> >>> >> extensions without using distutils. The problem of invoking the
> >>> >> compiler has been solved by many build systems and is not just a
> >>> >> unique and mysterious distutils feature.
> >>> >
> >>> > Did someone say it was? Building extensions is something distil does
> >>> > too, and
> >>> > without using distutils or setuptools.
> >>>
> >>> Right, the problem is the lack of a standard interface for how the
> >>> packaging system is supposed to invoke them - that is, *implementation
> >>> independent* docs of what the various setup.py commands are *supposed*
> >>> to do.
> >>>
> >>> The packaging system shouldn't have to care *how* setup.py is
> >>> implemented, but at the moment, the behaviour is underspecified to the
> >>> point where it's a matter of reverse engineering distutils and/or
> >>> implementing what seems necessary and waiting to see if people
> >>> complain.
> >>
> >>
> >> What are the plans for the build step in the grand plan of Python
> >> packaging? I think previously it has been suggested that once metadata
> is
> >> done and distribution/installation is taken care of the
> distutils/setuptools
> >> building part of all of this will be tackled. Is that still accurate?
> >
> > That's the priority list for my personal focus, but I certainly wouldn't
> > complain if there was momentum behind addressing the build side in
> parallel,
> > rather than in series. Commenting on PEPs is usually easier than writing
> > them, and the starting point for the build side is mostly just defining a
> > suitable subset of setup.py commands, options, behaviour and invocation
> > context that will make up the "fallback legacy interface" a new metabuild
> > system would need regardless of any other details.
> >
> > I just don't have the personal bandwidth to champion such an effort at
> this
> > point in time.
> >
> > Cheers,
> > Nick.
> >
> >
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> >
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140321/cb78d803/attachment-0001.html>