[SciPy-dev] PYTHONINCLUDE and sitecustomize.py

[Pearu Peterson]

On Thu, 19 Dec 2002 eric at localhost.localdomain wrote:

> Modified Files:
> 	build_clib.py build_ext.py 

Actually, build_clib.py does not need the PYTHONINCLUDE feature. I'd
remove it.

> Log Message:
> made changes to use a PYTHONINCLUDE environment variable to search for 
> include files during extension building.
> 
> (1) include directories.
> Distutils knows to include files from /usr/include/python2.2 (or
> wherever it is installed) whenever building extension modules.  
> Numeric installs its header files inside this directory when installed
> as root.  However, when I install Numeric in /home/eric/linux, the
> header files are installed in /home/eric/linux/python2.2.  Distutils
> doesn't know to look in hear for headers.  To solve this, I'd have to
> hack all the setup.py files for modules that rely on Numeric to use my
> include_dirs.  This isn't so nice.
> 
> To rectify the problem, scipy_distutils now searches for an
> environment variable called PYTHONINCLUDE that uses the same syntax as
> PYTHONPATH and PATH.  If PYTHONINCLUDE is found, the directories
> listed in it are added to the include path during compilation of C/C++
> files by build_ext.py and build_clib.py.  On my machine, I specify
> 
> export PYTHONINCLUDE=$OSDIR/include/python2.2
> 
> and all setup.py files work unaltered.  I think this is a reasonable
> solution.

There is one gotcha with the implementation of this solution. Consider
the following situation:

Numeric is installed as root. 
An user installs another version of Numeric using, say, --prefix=$HOME.
As a result Numeric header files are installed to 
$HOME/include/python2.2. 
Now, when building an extension module that uses Numeric and defining
PYTHONINCLUDE=$HOME/include/python2.2, then the compiler argument will
contain
  -I/usr/include/python2.2 -I$HOME/include/python2.2
As a result, wrong Numeric header files (from 
/usr/include/python2.2/Numeric) are being used when compiling extension
modules.

Obviously, $HOME/include/python2.2 should appear before
/usr/include/python2.2 and that can be easily accomplished by inserting
PYTHONINCLUDE directory(-ies) just before the '/usr/include/python2.2'
item in self.include_dirs list. It appears from distutils code that
'/usr/include/python2.2' is always the last item in the list and that
makes the fix easier.

However, IMHO distutils should take care of inserting 
<prefix>/include/python2.2 to the list include_dirs whenever
--prefix=<prefix> is used. And if this will be fixed in distutils, then
PYTHONINCLUDE feature becomes YAGNI.

So, I would avoid introducing PYTHONINCLUDE feature because the original
issue can be resolved using sitecustomize.py alone (after all that is the
"standard" way to customize Python). Moreover, the other issue of adding 
<prefix>/lib/python<version>/site-packages
(and .pth extensions) to sys.path can be resolved in sitecustomize.py as
well as Perry already indicated (and users would not need to modify their
PYTHONPATH).

A version of sitecustomize.py is attached to this message and I'd like to
propose adding this script to scipy directory.
The attached sitecustomize.py script contains also a fix to
distutils.util.get_platform function that is broken for Mac OSX system.
This script might be useful also for back-porting other Python
fixes/features to older Python versions that SciPy is supposed to support.

What do you think?

Pearu
-------------- next part --------------
#!/usr/bin/env python
#
# Purpose:
#   Facilitate installing/using Python extension modules to/from
#   non-standard locations when installation is carried out as
#     python setup.py install --prefix=$HOME
#   or
#     python setup.py install --prefix=$OSDIR
#   where the environment variable OSDIR contains arbitrary path.
#
# Recommended but optional environment (as defined in ~/.bashrc):
#  OSDIR=~/plat/`python -c 'import string;from distutils.util \
#           import get_platform;print string.replace(get_platform()," ","_")'`
#  export OSDIR
#  export PYTHONPATH=.:$HOME/site-python
#  export PATH=$OSDIR/bin:$PATH
#
# Usage:
#  Copy sitecustomize.py to location (e.g. to $HOME/site-python/)
#  where Python can import it.
#
# Features:
#  * Add <prefix>/lib/python<version>/site-packages to sys.path
#  * Add <prefix>/include/python<version> to the list of include_dirs
#    of distutils.build_ext.build_ext class
#  * Fix distutils.util.get_platform bug on Mac OS X
#
# Author: Pearu Peterson, Dec 2002
#

import os
import sys
import site
import string


########

from distutils.sysconfig import get_python_inc

l=len(sys.path)
prefixes = [os.environ.get('OSDIR'),os.environ.get('HOME')]
extra_include_dirs = {}

for prefix in prefixes:
    if prefix:
        if os.sep == '/':
            sitedirs = [os.path.join(prefix,
                                     "lib",
                                     "python" + sys.version[:3],
                                     "site-packages"),
                        os.path.join(prefix, "lib", "site-python")]
        else:
            sitedirs = [prefix, os.path.join(prefix, "lib", "site-packages")]
        for sitedir in sitedirs:
            if os.path.isdir(sitedir):
                site.addsitedir(sitedir)
        incdir = get_python_inc(prefix=prefix)
        if os.path.isdir(incdir):
            extra_include_dirs[incdir] = 1

extra_include_dirs = extra_include_dirs.keys()
sys.path = sys.path[l:] + sys.path[:l]

###########

from distutils.command import build_ext
old_build_ext = build_ext.build_ext
class site_build_ext(old_build_ext):
    def finalize_options (self):
        old_build_ext.finalize_options(self)
        l = len(self.include_dirs) - 1
        for d in extra_include_dirs:
            try:
                self.include_dirs.index(d)
            except ValueError:
                # insert just before system include_dir
                self.include_dirs.insert(l,d)
                l = l - 1
build_ext.build_ext = site_build_ext

###########

from distutils import util
old_get_platform = util.get_platform
def get_platform():
    return string.replace(old_get_platform()," ","_")
util.get_platform = get_platform

###########

if __name__=='__main__':
    site._test()