More than one interpreter per process?

[Roger Binns]

sturlamolden wrote:
> On 18 Des, 10:24, Roger Binns <rogerb at rogerbinns.com> wrote:
> 
>> The biggest stumbling block is what to do when the external environment
>> makes a new thread and then eventually calls back into Python.  It is
>> hard to know which interpretter that callback should go to.
> 
> Not if you explicitely hav to pass a pointer to the interpreter in
> every API call, which is what I suggested.

You missed my point.  What if the code calling back into Python doesn't
know which interpreter it belongs to?  Think of web server with python
callbacks registered for handling various things.  Currently that
situation works just fine as the simplified gil apis just pick the main
interpreter.

You have now imposed a requirement on all extension modules that they
need to keep track of interpreters in such a way that callbacks from new
threads not started by Python know which interpreter they belong to.
This is usually possible because you can give callback data in the
external environment apis, but your mechanism would prevent any that
don't have that ability from working at all.  We wouldn't find those
"broken" implementations until changing to your mechanism.

> But porting could be automated by a simple Python script. 

Have you actually tried it?  See if you can do it for the sqlite module
which is a standard part of the Python library.

> PEPs are closed for Python 3.

You glossed over my "prove the benefit outweighs the costs" bit :-)
This project will let you transparently use multiple processes:

 http://cheeseshop.python.org/pypi/processing

There are other techniques for parallelization using multiple processes
and even the network.  For example:

http://www.artima.com/forums/flat.jsp?forum=106&thread=214303
http://www.artima.com/weblogs/viewpost.jsp?thread=214235

Roger