[Distutils] Brewing controversy over header file installation

[Greg Ward]

Jack Jansen has been hacking on the Mac OS support in the Distutils, and
brings up an interesting -- but annoying point: installing
extension-related header files under Python's include directory could be
perceived as interfering with Python's source.  I disagree, but let me
see if I can summarize what's been going on in email between Jack,
Corran Webster (the other person working on Distutils Mac support), and
me.

I mentioned how files are installed on Mac OS (by quoting the source, of
course):
>      'mac': {
>          'purelib': '$base:Lib:site-packages',
>          'platlib': '$base:Lib:site-packages',
>          'headers': '$base:Include:$dist_name',
>          'scripts': '$base:Scripts',

Regarding the 'headers' install dir -- where extension-related header
files go -- Jack says:

> This one is wrong. Where do you put these things on Unix? I don't
> particularly like it if distutils scripts go mucking about in my
> source directories (which is what $base:Include would be).

On Unix, the answer is:

    'unix_prefix': {
        'purelib': '$base/lib/python$py_version_short/site-packages',
        'platlib': '$platbase/lib/python$py_version_short/site-packages',
        'headers': '$base/include/python$py_version_short/$dist_name',
        'scripts': '$base/bin',
        'data'   : '$base',
        },

In the "prefix" installation scheme, $base expands to sys.prefix, so in
a standard Python 2.0 installation on Unix, the 'headers' install dir is
/usr/local/include/python2.0/$dist_name.  $dist_name in turn expands to
the name of the Distribution -- "Numerical", "Imaging", whatever.

So yes, we are installing files *under* one of Python's directories, but
not *into* it -- the include/python2.0 directory continues to provide
just the header files installed by Python, namely Python.h and friends
(abstract.h, bitset.h, ..., unicodeobject.h).  And that's why I think
this isn't a bad thing to do.

The real reason it was done this way, of course, was laziness.  Any time
you compile a Python extension, you have to include
"-I/usr/local/include/python2.0" (or local equivalent).  With Numerical
Python's headers installed under this location, an extension that needs
access to Numerical's C structures just has to say
  #include <Numerical/arrayobject.h>

and it will Just Work.

Jack had an interesting suggestion:
> Moreover, for packages to use one another I would guess that more than just
> include files is needed (like the location of the shared libraries). I think
> that my preference (but I haven't thought about it very long) would be to put
> "distribution data for other distributions" all together in a single place,
> something like
>         $base/Extensions/packagename/Include
>         $base/Extensions/packagename/Lib

If the shared libraries that go in Extensions/$dist_name/Lib (Jack
misspelled it ;-) are C libraries, that's fine -- if they're Python
extensions, I'm confused.

This makes sense, but it does require adding another directory to the
standard include search path when building Python extensions.  And the
directory would probably be named something different on Unix.  But it's
in interesting idea, and I'm not going to throw it out willy-nilly.
What does everyone else think?

Corran's response to this was:
> I agree, though, that this would be a natural way to go.  It might
> make more sense to have it all inside the site-packages directory,
> though: headers live in
> 
> $base:Lib:site-packages:$package:Include
> 
> while modules (both python and C shared libraries) live in
> 
> $base:Lib:site-packages:$package:

I disagree: site-packages should be for Python modules and extensions.
If we have to install C headers and libraries -- and I suspect we
probably do -- then they should find their own home.  On Unix, perhaps
another directory under <exec-prefix>/lib/python2.0, but I'm not sure
what to call it that makes it clear these are for *C* programming, not
Python programming.  (Granted it's the specialized domain of C
programming for developing Python extensions, but still.)

On Mac OS and Windows, it's easy -- I'd say put 'em in
Extensions/Includes and Extensions/Lib under <prefix> (where that / is
really a : or a \, of course).

Jack later follows up:
> Ok, so then we have (assuming we're running in a source tree) an "Include"
> which contains standard Python include files and "includes" which contains
> package-deposited include files. I'm not too thrilled with the similar names,
> but I guess I can live with this.

Yechh -- I would be *very* wary of depending on case distinction in
filenames!  I still don't see what's so wrong with Include/*.h being
standard Python headers, and Include/*/*.h being headers provided by
whatever extensions are installed on the system.

        Greg
-- 
Greg Ward                                      gward@python.net
http://starship.python.net/~gward/