[issue32797] Tracebacks from Cython modules no longer work

[Nick Coghlan]

Nick Coghlan <ncoghlan at gmail.com> added the comment:

While I'd be inclined to agree with Paul's analysis if CPython were a greenfield project, I also think if SageMath had already been ported to Python 3.2, we would have considered this change a behavioural regression between 3.2 and 3.3 resulting from the importlib migration and worked out a way to get it to work implicitly again.

Now, though, we need a way for SageMath to get it working on releases up to and including 3.7, *without* any help from Cython or CPython, since it needs to work with existing Cython releases on existing CPython versions to avoid presenting itself to SageMath users as regression introduced by switching from Python 2 to Python 3.

To do that, I believe it can be made to work in much the same way it did in Python 2 if SageMath were to do the following:

1. Define a subclass of ExtensionModuleLoader [1] that overrides get_source() to also look for a ".pyx" file adjacent to the extension module. This would also look for any of the file suffixes in SOURCE_SUFFIXES if a .pyx file isn't found.
2. Create an instance of FileFinder [3] that uses the custom loader subclass for any of the file suffixes in EXTENSION_SUFFIXES [4]
3. Replace the regular file finding hook in sys.path_hooks with  the path_hook method from that new FileFinder instance (by default, there's only one FileFinder hook installed, and it can currently be identified as "obj.__name__ == 'path_hook_for_FileFinder')
4. Invalidate importlib's caches, so any future extension module imports will use the custom extension module loader, rather than the default one

[1] https://docs.python.org/3/library/importlib.html#importlib.machinery.ExtensionFileLoader
[2] https://docs.python.org/3/library/importlib.html#importlib.machinery.SOURCE_SUFFIXES
[3] https://docs.python.org/3/library/importlib.html#importlib.machinery.FileFinder
[4] https://docs.python.org/3/library/importlib.html#importlib.machinery.EXTENSION_SUFFIXES

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue32797>
_______________________________________