2.6, 3.0, and truly independent intepreters

[Thomas Heller]

Andy schrieb:
> Dear Python dev community,
> 
> [...]  Basically,
> we use embedded python and use it to wrap our high performance C++
> class set which wraps OpenGL, DirectX and our own software renderer.
> In addition to wrapping our C++ frameworks, we use python to perform
> various "worker" tasks on worker thread (e.g. image loading and
> processing).  However, we require *true* thread/interpreter
> independence so python 2 has been frustrating at time, to say the
> least.
[...]
> 
> Sadly, the only way we could get truly independent interpreters was to
> put python in a dynamic library, have our installer make a *duplicate*
> copy of it during the installation process (e.g. python.dll/.bundle ->
> python2.dll/.bundle) and load each one explicitly in our app, so we
> can get truly independent interpreters.  In other words, we load a
> fresh dynamic lib for each thread-independent interpreter (you can't
> reuse the same dynamic library because the OS will just reference the
> already-loaded one).

Interesting questions you ask.

A random note: py2exe also does something similar for executables build
with the 'bundle = 1' option.  The python.dll and .pyd extension modules
in this case are not loaded into the process in the 'normal' way (with
some kind of windows LoadLibrary() call, instead they are loaded by code
in py2exe that /emulates/ LoadLibrary - the code segments are loaded into
memory, fixups are made for imported functions, and marked executable.

The result is that separate COM objects implemented as Python modules and
converted into separate dlls by py2exe do not share their interpreters even
if they are running in the same process.  Of course this only works on windows.
In effect this is similar to using /statically/ linked python interpreters
in separate dlls.  Can't you do something like that?

> So, my question becomes: is python 3 ready for true multithreaded
> support??  Can we finally abandon our Frankenstein approach of loading
> multiple identical dynamic libs to achieve truly independent
> interpreters?? I've reviewed all the new python 3 C API module stuff,
> and all I have to say is: whew--better late then never!!  So, although
> that solves modules offering truly independent interpreter support,
> the following questions remain:
> 
> - In python 3, the C module API now supports true interpreter
> independence, but have all the modules in the python codebase been
> converted over?  Are they all now truly compliant?  It will only take
> a single static/global state variable in a module to potentially cause
> no end of pain in a multiple interpreter environment!  Yikes!

I don't think this is the case (currently).  But you could submit patches
to Python so that at least the 'official' modules (builtin and extensions)
would behave corectly in the case of multiple interpreters.  At least
this is a much lighter task than writing your own GIL-less interpreter.

My 2 cents,

Thomas