is modulefinder.ModuleFinder working at all?

[Steve D'Aprano]

On Thu, 10 Nov 2016 08:08 am, Wolfgang Maier wrote:

> Hi,
> 
> I just used the stdlib's modulefinder.ModuleFinder (intended to find
> modules used by a script) for the first time in my life and it just
> doesn't seem to work like documented at all.
> Not sure what is going on, but if I try the usage example from
> https://docs.python.org/3/library/modulefinder.html
> it's reporting every single module from the stdlib whether imported or
> not! 

I see what you mean, but it's not quite *every* module. After I run the
example from the docs, I get 197 modules, and here's at least two that
aren't included:

py> len(finder.modules)
197
py> 'statistics' in finder.modules
False
py> 'cmath' in finder.modules
False


Curiously, it includes modules that aren't cached in sys.modules:

py> len(finder.modules.keys() - sys.modules.keys())
107


So I'm not sure how that's possible. 

According to the example module, it imports re and itertools. Both of those
have already been imported, and so will be cached in sys.modules, as will
all their dependencies. So I don't see how it is possible that ModuleFinder
can find dependencies that aren't cached.

Theoretically, of course some dependency might import a bunch of modules,
then delete them from sys.modules. If it were only one or two, I'd believe
that. But not 107 of them.

After running the module finder on the sample file, I ran the report()
method to get a nicer display of the imported modules:

py> finder = ModuleFinder()
py> finder.run_script('/tmp/bacon.py')
py> finder.report()

  Name                      File
  ----                      ----
m __future__                /usr/local/lib/python3.5/__future__.py
m __main__                  /tmp/bacon.py
m _ast
m _bootlocale               /usr/local/lib/python3.5/_bootlocale.py
m _bz2                      /usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-i386-linux-gnu.so
[...]


which shows the unittest package being loaded, which is pretty dubious.


On the other hand, here's a simpler example which seems to work fine:


py> with open('/tmp/do_little.py', 'w') as f:
...     f.write('import math\n')
...
12
py> finder = ModuleFinder()
py> finder.run_script('/tmp/do_little.py')
py> finder.report()

  Name                      File
  ----                      ----
m __main__                  /tmp/do_little.py
m math                      /usr/local/lib/python3.5/lib-dynload/math.cpython-35m-i386-linux-gnu.so


So I'm not really sure what's going on.





-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.