is modulefinder.ModuleFinder working at all?

[Wolfgang Maier]

On 10.11.2016 01:02, Steve D'Aprano wrote:
> On Thu, 10 Nov 2016 08:08 am, Wolfgang Maier wrote:
>
>> Hi,
>>
>> I just used the stdlib's modulefinder.ModuleFinder (intended to find
>> modules used by a script) for the first time in my life and it just
>> doesn't seem to work like documented at all.
>> Not sure what is going on, but if I try the usage example from
>> https://docs.python.org/3/library/modulefinder.html
>> it's reporting every single module from the stdlib whether imported or
>> not!
>
> I see what you mean, but it's not quite *every* module. After I run the
> example from the docs, I get 197 modules, and here's at least two that
> aren't included:
>
> py> len(finder.modules)
> 197
> py> 'statistics' in finder.modules
> False
> py> 'cmath' in finder.modules
> False
>
>
> Curiously, it includes modules that aren't cached in sys.modules:
>
> py> len(finder.modules.keys() - sys.modules.keys())
> 107
>
>
> So I'm not sure how that's possible.
>
> According to the example module, it imports re and itertools. Both of those
> have already been imported, and so will be cached in sys.modules, as will
> all their dependencies. So I don't see how it is possible that ModuleFinder
> can find dependencies that aren't cached.
>
> Theoretically, of course some dependency might import a bunch of modules,
> then delete them from sys.modules. If it were only one or two, I'd believe
> that. But not 107 of them.
>
> After running the module finder on the sample file, I ran the report()
> method to get a nicer display of the imported modules:
>
> py> finder = ModuleFinder()
> py> finder.run_script('/tmp/bacon.py')
> py> finder.report()
>
>   Name                      File
>   ----                      ----
> m __future__                /usr/local/lib/python3.5/__future__.py
> m __main__                  /tmp/bacon.py
> m _ast
> m _bootlocale               /usr/local/lib/python3.5/_bootlocale.py
> m _bz2                      /usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-i386-linux-gnu.so
> [...]
>
>
> which shows the unittest package being loaded, which is pretty dubious.
>
>
> On the other hand, here's a simpler example which seems to work fine:
>
>
> py> with open('/tmp/do_little.py', 'w') as f:
> ...     f.write('import math\n')
> ...
> 12
> py> finder = ModuleFinder()
> py> finder.run_script('/tmp/do_little.py')
> py> finder.report()
>
>   Name                      File
>   ----                      ----
> m __main__                  /tmp/do_little.py
> m math                      /usr/local/lib/python3.5/lib-dynload/math.cpython-35m-i386-linux-gnu.so
>
>
> So I'm not really sure what's going on.
>

Thanks for the detailed analysis.
You are right that it seems to work with certain imports. I verified 
your example with import math and found a few additional examples.

Things seem to work with imports of any of the following (alone or in 
combination):
io, itertools, math, sys

OTOH, I doubt that it reports correctly with:
os, heapq, operator, re

Personally, I fail to see a pattern here, but maybe somebody else can 
explain that behaviour.
I'd really like to report this to the bug tracker, but at the moment I'm 
not sure as what exactly. It seems that at least the documentation is 
incorrect since the usage example there is not working.

Best,
Wolfgang