Multiple python interpreters within the same process

[Graham Dumpleton]

On Jun 10, 9:07 am, Josiah Carlson <josiah.carl... at sbcglobal.net>
wrote:
> André wrote:
> > On Jun 9, 5:00 pm, "Marcin Kalicinski" <kal... at poczta.onet.pl> wrote:
> >> How do I use multiple Python interpreters within the same process?
>
> >> I know there's a function Py_NewInterpreter. However, how do I use functions
> >> like Py_RunString etc. with it? They don't take any arguments that would
> >> tell on which interpreter to run the string...?
>
> >> Marcin
>
> > You may want to look at the code modulehttp://docs.python.org/lib/module-code.html
>
> That's completely unrelated.
>
> To answer Marcin's question, from what I understand, running multiple
> Python interpreters is not supported.  There are various issues with
> object sharing and refcounts, etc., and are unlikely to be fixed soon if
> ever.

I really don't know why people keep propagating this myth that you
can't run multiple Python interpreters. :-(

The Python C API has supported multiple Python interpreter instances
within a single process for a long time. If you write your C program
correctly there isn't a problem. The only real limitation is that
different Python interpreter instances can't use different versions of
a C extension module as they when loaded are shared across all Python
interpreter instances.

C extension modules can also cause other problems as well, but this
isn't the fault of Python but of the module writers. Specifically, if
a C extension module is written to only use the simplified API for GIL
locking it can only be used with the first Python interpreter instance
created. If they use the wider GIL locking API properly then there is
no problem with using it in other Python interpreter instances.
Another issue although one which you aren't likely to come across
unless you are doing really tricky stuff, is where a C extension
module was written so as to assume that once it is loaded into a
Python interpreter instance, that that interpreter will not be
destroyed. From what I have seen Pyrex generated C extension modules
possibly have this latter problem and will cause a process to crash if
a Python interpreter instance is destroyed and then a new one created
in its place.

As proof that all this stuff can work, have a look at mod_python and
mod_wsgi. Both make use of multiple Python interpreter instances in a
single process. The mod_wsgi documentation even points out the above
problems with C extension modules in latter sections of:

  http://code.google.com/p/modwsgi/wiki/ApplicationIssues

Graham