[issue45020] Freeze all modules imported during startup.

[Gregory Szorc]

Gregory Szorc <gregory.szorc at gmail.com> added the comment:

When I investigated freezing the standard library for PyOxidizer, I ran into a rash of problems. The frozen importer doesn't behave like PathFinder. It doesn't (didn't?) set some common module-level attributes that are documented by the importer "specification" to be set and this failed a handful of tests and lead to runtime issues or breakage in 3rd party packages (such as random packages looking for a __file__ on a common stdlib module).

Also, when I last looked at the CPython source, the frozen importer performed a linear scan of its indexed C array performing strcmp() on each entry until it found what it was looking for. So adding hundreds of modules could result in sufficient overhead and justify using a more efficient lookup algorithm. (PyOxidizer uses Rust's HashMap to index modules by name.)

I fully support more aggressive usage of frozen modules in the standard library to speed up interpreter startup. However, if you want to ship this as enabled by default, from my experience with PyOxidizer, I highly recommend:

* Make sure you run unit tests against the frozen modules. If you don't do this, subtle differences in how the different importers behave will lead to problems.
* Shoring up the implementation of the frozen importer to make it better conform with the importer specification.
* Be prepared for people to complain about the loss of __file__. e.g. https://github.com/indygreg/PyOxidizer/issues/69 / https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_behavior_and_compliance.html?highlight=__file__#file-and-cached-module-attributes

----------
nosy: +indygreg

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45020>
_______________________________________