[New-bugs-announce] [issue45703] importlib.invalidate_caches() does not invalidate _NamespacePath's _last_parent_path-based cache
Miro Hrončok
report at bugs.python.org
Wed Nov 3 07:21:56 EDT 2021
New submission from Miro Hrončok <miro at hroncok.cz>:
Recently, when debugging a weird problem (see https://bugzilla.redhat.com/show_bug.cgi?id=2018551 for details if interested, but not important for this issue), I've realized that the _NamespacePath class (from importlib/_bootstrap_external.py) has a cache that (in some cases) uses tuple(sys.path) as the key. I was expecting importlib.invalidate_caches() to invalidate this cache, but it doesn't.
Consider the following directory structure:
.
├── PATH1
│ └── namespace
│ └── sub1
│ └── __init__.py
└── PATH2
└── namespace
└── sub2
└── __init__.py
Here is a helper to create it (on Linux-ish):
$ mkdir -p PATH1/namespace/sub1
$ mkdir -p PATH2/namespace/sub2
$ touch PATH1/namespace/sub1/__init__.py
$ touch PATH2/namespace/sub2/__init__.py
Run Python with PYTHONPATH=PATH1:PATH2 (output slightly formatted for readability):
$ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace',
'.../namespace_path_cache/PATH2/namespace'])
>>> import namespace.sub1 # works
>>> import namespace.sub2 # works
>>> exit()
The namespace packages seem to work as expected.
Now move PATH2/namespace out of the way:
$ mv PATH2/namespace PATH2/cant-import-this
Run Python again:
$ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> ...
While this interpreter still runs, move the PATH2/namespace module back in:
$ mv PATH2/cant-import-this PATH2/namespace
>>> ...
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'
>>> import importlib
>>> importlib.invalidate_caches() # invalidate the cache, not helpful
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'
>>> import sys
>>> sys.path.remove('') # changing sys.path solves this
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace',
'.../namespace_path_cache/PATH2/namespace'])
importlib.invalidate_caches() documentation says:
> This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.
That makes me think calling importlib.invalidate_caches() should also invalidate the cache of _NamespacePaths.
(This also affects older Pythons, but since it is a behavior change, I've only marked 3.11).
----------
messages: 405606
nosy: hroncok, petr.viktorin
priority: normal
severity: normal
status: open
title: importlib.invalidate_caches() does not invalidate _NamespacePath's _last_parent_path-based cache
type: behavior
versions: Python 3.11
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue45703>
_______________________________________
More information about the New-bugs-announce
mailing list