[Distutils] pip can't find header file for extension module, but `python setup.py install` works fine

[Robert Collins]

On 2 June 2015 at 02:51, Erik Bray <erik.m.bray at gmail.com> wrote:
> On Sun, May 31, 2015 at 5:07 PM, AJ Friend <ajfriend at gmail.com> wrote:
>> Hi,
>>
>> I'm trying to write a new `setup.py` file for an extension module to
>> wrap a C library (https://github.com/cvxgrp/scs).
>>
>> The current `setup.py` file imports numpy. I'm trying to delay that
>> import statement until setuptools has a chance to install numpy if
>> it's not already installed. I'm trying to do that with this bit of
>> code:
>>
>> from setuptools.command.build_ext import build_ext as _build_ext
>> class build_ext(_build_ext):
>>     def finalize_options(self):
>>         _build_ext.finalize_options(self)
>>         # Prevent numpy from thinking it is still in its setup process:
>>         __builtins__.__NUMPY_SETUP__ = False
>>         import numpy
>>         self.include_dirs += ext['include_dirs'] + [numpy.get_include()]
>>
>> Running `python setup.py install` seems to work fine on my OSX
>> machine, but when I run `pip install .` in the directory with
>> `setup.py`, I get a clang error that it can't find one of the header
>> files.
>>
>> Any idea why that would be happening? Could it have anything to do
>> with the relative path I'm giving for the include directories?
>>
>> Also, I had trouble finding good documentation on subclassing
>> build_ext. Does anyone know if setting self.include_dirs overwrites or
>> appends to the include_dirs attribute of an Extension object defined
>> later in setup.py?
>>
>> For the curious, my current attempt at setup.py is
>> athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The
>> original can be found in the same directory.
>>
>> More generally, since I'm new to python packaging, I'm not sure how
>> well or correctly I've written my `setup.py` file. Any feedback on
>> doing things correctly would be appreciated.
>
> Hi AJ,
>
> For a lot of things in Python packaging there is not, sadly, One Right
> Way to Do It.  Your setup.py looks okay though.
>
> You may want to have a look at the get_numpy_include_path utility here:
> https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66
>
> It's similar to what you're already doing, but maybe a little more
> 'robust'.  In particular, I think the reload of the numpy module may
> be important.

It is, because what numpy is doing is importing from its own tree
before setup has completed - which is in general unsafe.

And numpy may be in process depending on exactly what setuptools
setup_requires easy-install code has done, which can lead to the
symptoms you see. This should be better once we get pip interpreting
setup_requires for you, but a perhaps shorter term fix would be to
factor out the distutils support glue from numpy (which seems pretty
self contained) into a out of package file - either adjacent to
setup.py, or, if numpy is willing to make the dependency on setuptools
a hard dep (which I recommend), then as a separate package which can
be setup_require'd.

That would avoid the bootstrapping complexity, resulting in numpy not
being imported until it has actually been installed, and things should
be rosy for folk downstream of numpy from that point on.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud