[Python-Dev] Sub-interpreters: importing numpy causes hang

[Gregory P. Smith]

On Thu, Jan 24, 2019 at 1:25 PM Chris Barker - NOAA Federal via Python-Dev <
python-dev at python.org> wrote:

> If your primary concern is module clashes between plugins, maybe you
> can hack around that:
>
> 1) if the plugins are providing copies of any other modules, then you
> can simply require them to put them in their own namespace — that is,
> a plug-in is a single package, with however many sub modules as it may
> need.
>
> 2) if plugins might require third party packages that need to be
> isolated, then maybe you could use an import hook that
> re-names/isolates the modules each plugin loads, so they are kept
> separate.
>
> I haven’t thought through how to do any of this, but in principle, you
> can have the same module loaded twice if it has a different name.
>

This is dangerous for extension modules.  C is a single global space
unrelated to Python module names that cannot be isolated without
intentionally building and linking each desired extension module statically
and configured not to export its own symbols (no-export-dynamic).  Non
trivial.

Suggesting importing the same extension module multiple times under
different Python sys.modules names is a recipe for disaster.  Most
extension module code is not written with that in mind.  So while *some*
things happen to "work", many others blow up in unexpected hard to debug
ways.

Not that sub interpreters aren’t cool and useful, but you can probably
> handle module clashes in a simpler way.
>

They're a cool and useful theory... but I really do not recommend their use
for code importing other libraries expecting to be isolated.  CPython
doesn't offer multiple isolated runtimes in a process today.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190127/f60b3335/attachment.html>