[issue25533] Make pkgutil.iter_modules() yield built-in modules

[Martin Panter]

Martin Panter added the comment:

I did some work on adding support for frozen modules, but I got stuck. The low level is fine:

>>> pprint(sys.get_frozen_modules())  # (name, ispkg) pairs
(('_frozen_importlib', False),
 ('_frozen_importlib_external', False),
 ('__hello__', False),
 ('__phello__', True),
 ('__phello__.spam', False))
>>> print("\n".join(map(repr, pkgutil.iter_modules(builtins=True))))
(<class '_frozen_importlib.BuiltinImporter'>, '_ast', False)
. . .
(<class '_frozen_importlib.BuiltinImporter'>, 'zipimport', False)
(<class '_frozen_importlib.FrozenImporter'>, '_frozen_importlib', False)
(<class '_frozen_importlib.FrozenImporter'>, '_frozen_importlib_external', False)
(<class '_frozen_importlib.FrozenImporter'>, '__hello__', False)
(<class '_frozen_importlib.FrozenImporter'>, '__phello__', True)
. . .
(FileFinder('.'), 'python-config', False)
. . .

But the current __hello__ and __phello__ modules print stuff when you import them, which messes with walk_packages(), pydoc, etc:

$ ./python -m pydoc -k pkgutil
Hello world!
Hello world!
Hello world!
pkgutil - Utilities to support packages.
test.test_pkgutil 

When I stopped these frozen modules from printing on import (as in my current patch), I found this broke the test suite. In particular, test_importlib.frozen.test_loader relies on the printouts to test what gets executed when importing frozen modules. So I am not sure the best way to continue if I am to add support for frozen modules to iter_modules().

Another problem was that there is no way to list submodules of a frozen package unless you know the package’s name. Currently, iter_modules() only sees a path list, which is empty for __phello__. However, I was able to add a special case to walk_packages(None) to include frozen submodules.

Some questions:

1. Do people think the general idea of enhancing iter_modules() is worthwhile?

2. Should I try harder to solve the problem with running frozen modules, or is it sensible to just leave out the frozen module support?

----------
Added file: http://bugs.python.org/file41072/iter-builtin-frozen.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25533>
_______________________________________