[Numpy-discussion] numpy distutils log error with easy_install

[Matthew Brett]

Hi,

On Thu, Oct 4, 2012 at 10:29 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Thu, Oct 4, 2012 at 7:35 AM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>
>>
>> On Thu, Oct 4, 2012 at 12:14 AM, Matthew Brett <matthew.brett at gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> On Wed, Oct 3, 2012 at 10:51 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>> wrote:
>>> >
>>> >
>>> > On Wed, Oct 3, 2012 at 11:29 PM, Matthew Brett <matthew.brett at gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> On Wed, Oct 3, 2012 at 10:17 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>> >> wrote:
>>> >> >
>>> >> >
>>> >> > On Mon, Oct 1, 2012 at 10:47 PM, Matthew Brett
>>> >> > <matthew.brett at gmail.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> On Mon, Oct 1, 2012 at 9:42 PM, Matthew Brett
>>> >> >> <matthew.brett at gmail.com>
>>> >> >> wrote:
>>> >> >> > Hi,
>>> >> >> >
>>> >> >> > One of our kind users pointed out an error when using easy_install
>>> >> >> > to
>>> >> >> > install our package nipy.   I've reproduced it now on a bare
>>> >> >> > package
>>> >> >> > using numpy distutils and having a trivial extension:
>>> >> >> >
>>> >> >> > https://github.com/matthew-brett/apkg
>>> >> >> >
>>> >> >> > To reproduce:
>>> >> >> >
>>> >> >> > git clone git://github.com/mathew-brett/apkg.git
>>> >> >> >
>>> >> >> > easy_install apkg
>>> >> >> >
>>> >> >> > You should get something like this:
>>> >> >> >
>>> >> >> > Processing apkg
>>> >> >> > Running setup.py -q bdist_egg --dist-dir
>>> >> >> > /home/mb312/tmp/apkg/egg-dist-tmp-T5yjuB
>>> >> >> > Appending apkg configuration to
>>> >> >> > Ignoring attempt to set 'name' (from '' to 'apkg')
>>> >> >> > zip_safe flag not set; analyzing archive contents...
>>> >> >> > Adding apkg 0.1 to easy-install.pth file
>>> >> >> >
>>> >> >> > Installed
>>> >> >> >
>>> >> >> >
>>> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/apkg-0.1-py2.6-linux-i686.egg
>>> >> >> > Processing dependencies for apkg==0.1
>>> >> >> > Finished processing dependencies for apkg==0.1
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > /home/mb312/.virtualenvs/np-1.6.2/lib/python2.6/site-packages/numpy/distutils/misc_util.py:252:
>>> >> >> > RuntimeWarning: Parent module 'numpy.distutils' not found while
>>> >> >> > handling absolute import
>>> >> >> >   from numpy.distutils import log
>>> >> >> >
>>> >> >> > Note the last error.
>>> >> >>
>>> >> >> Sorry, correcting myself - it's (obviously) a Warning rather than an
>>> >> >> error, but still distracting, and it would be good to avoid it if
>>> >> >> possible...
>>> >> >
>>> >> >
>>> >> > The combination of two or all of atexit.register, easy_install and
>>> >> > virtualenv seems to be causing this. Unless someone feels like
>>> >> > digging
>>> >> > into
>>> >> > that (I certainly don't), there are two easy solutions:
>>> >> > 1. Silence the warning.
>>> >>
>>> >> Sorry - I am not sure what you mean.  The problem here is the user who
>>> >> assumes that something bad happened when running easy_install - which
>>> >> is what happened in the case of nipy.  Is there some way of silencing
>>> >> this (specific) warning from within setup.py?
>>> >>
>>> >> > 2. Remove the offending import and the logging. This will only remove
>>> >> > the
>>> >> > line "removing: _configtest.c _configtest.o" from the build log
>>> >> > (x20).
>>> >>
>>> >> Which import did you mean?  I think I need all the imports I'm using
>>> >> in the example minimal package.  I'm not explicitly importing logging
>>> >> for example.
>>> >
>>> >
>>> > The import that's indicated in the warning, on line 252 of
>>> > numpy/distutils/misc_util.py. Relevant code:
>>> >
>>> > _temporary_directory = None
>>> > def clean_up_temporary_directory():
>>> >     from numpy.distutils import log    # CAUSES RUNTIME WARNING
>>> >     global _temporary_directory
>>> >     if not _temporary_directory:
>>> >         return
>>> >     log.debug('removing %s', _temporary_directory)
>>> >     try:
>>> >         shutil.rmtree(_temporary_directory)
>>> >     except OSError:
>>> >         pass
>>> >     _temporary_directory = None
>>> >
>>> > def make_temp_file(suffix='', prefix='', text=True):
>>> >     global _temporary_directory
>>> >     if not _temporary_directory:
>>> >         _temporary_directory = tempfile.mkdtemp()
>>> >         atexit.register(clean_up_temporary_directory)
>>>
>>> Sorry - I still don't understand.  You mean I should (in my package -
>>> say nipy or 'apkg') monkey-patch numpy distutils.misc_util ?
>>
>>
>> No, I was proposing to fix this in numpy.distutils.misc_util directly. Sorry
>> for not being clear.
>>
>>>
>>>
>>> Another option would be to move the import outside the callback function
>>> thus:
>>
>>
>> Should work too I guess, but since I don't understand why the import is
>> where it is now I'm not 100% sure. Removing the import on line 252 and the
>> log.debug call would be the safest fix.
>
> The import is in the function because the following outside the
> function (at the top of misc_util):
>
> from numpy.distutils import log
>
> leads to the following on `python setup.py install`:
>
> Running from numpy source directory.
> Traceback (most recent call last):
>   File "setup.py", line 214, in <module>
>     setup_package()
>   File "setup.py", line 191, in setup_package
>     from numpy.distutils.core import setup
>   File "/home/mb312/dev_trees/numpy/numpy/distutils/__init__.py", line
> 7, in <module>
>     import ccompiler
>   File "/home/mb312/dev_trees/numpy/numpy/distutils/ccompiler.py",
> line 14, in <module>
>     from numpy.distutils import log
>   File "/home/mb312/dev_trees/numpy/numpy/distutils/log.py", line 9, in <module>
>     from misc_util import red_text, default_text, cyan_text,
> green_text, is_sequence, is_string
>   File "/home/mb312/dev_trees/numpy/numpy/distutils/misc_util.py",
> line 21, in <module>
>     from numpy.distutils import log
> ImportError: cannot import name log
>
> I'm guessing this is a problem of circular imports, because in
> numpy.distutils.log we see that numpy distutils is importing
> misc_util.
>
> It so happens that:
>
> from numpy.distutils.log import debug as log_debug
>
> works in this situation, I don't understand why.  I suppose also that
> it may rely on something complex and / or undocumented in the details
> of python's import mechanism, and so might be better avoided.
>
> So, is the 'remove log call in function' reasonable for a patch to
> trunk?  For the upcoming release?

Ralf's suggestion as pull request:

https://github.com/numpy/numpy/pull/480

Best,

Matthew