[Python-ideas] In-process interpreters

[Swapnil Talekar]

I am working on a project where we have a very peculiar requirements for an
embedded interpreter. In my opinion these requirements cannot be extremely
rare. I am posting this message to get some feedback from people who have
worked on similar stuff or are interested in such a solution. If such a
solution would be useful to a lot people if readily available as open
source.

We need to run multiple interpreters in-process. Which means we need to
create completely independent environments for each interpreter using some
very dirty hack. Since Python inherently does not support this. Python
threads( created using python's threading module ) are not of much help
because of the GIL.
For this purpose the solution we have adopted is moving all the mutable
python globals to heap.

Nokia did a similar change to Python while porting it to S60 platform.
Reference <http://research.nokia.com/files/PythonOnAPhone.ppt>
It seems Nokia has stopped using this solution but I am not aware of how are
they going about now since I cannot think of any other solution to run
multiple in-process interpreters. Jukka, can you throw some light ?

Lua already supports this.

I would expect other projects like mod_python would also have to face
similar problem but I am not aware how they are handling it.

People who have ported Python on flat-memory model OSes such as VxWorks
surely face this problem (Since there are no processes) unless they don't
have requirement of running multiple interpreters.

The second major change we did is to reinitialize python after executing of
each script. This is because of the limitation of running it on embedded
platform hardware and limited memory constraints. Again Python inherently
does not provide support for this. (Calling Py_Initialize and
Py_Finalize repetitively is not enough. It does not completely reinitialize
python and is also known to leak memory)


Would it be worthwhile (benefit a lot of people ) if these changes be made
to standard python (eg. http://bugs.python.org/issue3329 ). Possibly a new
branch of Python which supports multiple in-process interpreters?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20101117/7628d699/attachment.html>