[New-bugs-announce] [issue40350] modulefinder chokes on numpy - dereferencing None in spec.loader

[Greg Whiteley]

New submission from Greg Whiteley <greg.whiteley at gmail.com>:

Issue:

Running ModuleFinder.run_script() on numpy versions 1.16.1 to 1.18.3 (maybe more) fails with backtrace.  See steps to reproduce below.

I do not see this problem on earlier versions of python than 3.8 (tested 3.4, 3.5, 3.6 on ubuntu LTSs), but the code has changed around 3.8.

The failure comes to this line of modulefinder.py

https://github.com/python/cpython/blame/master/Lib/modulefinder.py#L79

    if spec.loader.is_package(name):
        return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)

I can work around it by changing that to check for None

    if spec.loader is not None and spec.loader.is_package(name):
        return None, os.path.dirname(file_path), ("", "", _PKG_DIRECTORY)


Environment:

Ubuntu 20.04 with default python3, python3-pip

Steps to reproduce:

# note any nump version I've tried 1.16.1 and greater fails - I included 1.18.3 to be precise for reproduciton
$ pip3 install "numpy==1.18.3"
$ cat test.py
import numpy

$ python3
>>> from modulefinder import ModuleFinder
>>> finder = ModuleFinder()
>>> finder.run_script("test.py")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/modulefinder.py", line 165, in run_script
    self.load_module('__main__', fp, pathname, stuff)
...

300 lines of stack elided - see attached fulllog.txt

...
  File "/usr/lib/python3.8/modulefinder.py", line 433, in scan_code
    self._safe_import_hook(name, m, fromlist, level=0)
  File "/usr/lib/python3.8/modulefinder.py", line 378, in _safe_import_hook
    self.import_hook(name, caller, level=level)
  File "/usr/lib/python3.8/modulefinder.py", line 177, in import_hook
    q, tail = self.find_head_package(parent, name)
  File "/usr/lib/python3.8/modulefinder.py", line 233, in find_head_package
    q = self.import_module(head, qname, parent)
  File "/usr/lib/python3.8/modulefinder.py", line 320, in import_module
    fp, pathname, stuff = self.find_module(partname,
  File "/usr/lib/python3.8/modulefinder.py", line 511, in find_module
    return _find_module(name, path)
  File "/usr/lib/python3.8/modulefinder.py", line 78, in _find_module
    if spec.loader.is_package(name):
AttributeError: 'NoneType' object has no attribute 'is_package'
>>> 


Obviously I can't tell if numpy or modulefinder is the real culprit.

Let me know if I can give any more information.

----------
components: Library (Lib)
files: fulllog.txt
messages: 366912
nosy: Greg Whiteley
priority: normal
severity: normal
status: open
title: modulefinder chokes on numpy - dereferencing None in spec.loader
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file49079/fulllog.txt

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40350>
_______________________________________