Embedding multiple interpreters

[Garthy]

Hi Chris (and Michael),

On 06/12/13 15:51, Chris Angelico wrote:
> On Fri, Dec 6, 2013 at 4:16 PM, Michael Torrie<torriem at gmail.com>  wrote:
>> On 12/05/2013 07:34 PM, Garthy wrote:
>>> - My fallback if I can't do this is to implement each instance in a
>>> dedicated *process* rather than per-thread. However, there is a
>>> significant cost to doing this that I would rather not incur.
>>
>> What cost is this? Are you speaking of cost in terms of what you the
>> programmer would have to do, cost in terms of setting things up and
>> communicating with the process, or the cost of creating a process vs a
>> thread?  If it's the last, on most modern OS's (particularly Linux),
>> it's really not that expensive.  On Linux the cost of threads and
>> processes are nearly the same.
>
> If you want my guess, the cost of going to multiple processes would be
> to do with passing data back and forth between them. Why is Python
> being embedded in another application? Sounds like there's data moving
> from C to Python to C, ergo breaking that into separate processes
> means lots of IPC.

An excellent guess. :)

One characteristic of the application I am looking to embed Python in is 
that there are a fairly large number calls from the app into Python, and 
for each, generally many back to the app. There is a healthy amount of 
data flowing back and forth each time. An implementation with an 
inter-process roundtrip each time (with a different scripting language) 
proved to be too limiting, and needlessly complicated the design of the 
app. As such, more development effort has gone into making things work 
better with components that work well running across thread boundaries 
than process boundaries.

I am confident at this point I could pull things off with a Python 
one-interpreter-per-process design, but I'd then need to visit the IPC 
side of things again and put up with the limitations that arise. 
Additionally, the IPC code has has less attention and isn't as capable. 
I know roughly how I'd proceed if I went with this approach, but it is 
the least desirable outcome of the two.

However, if I could manage to get a thread-based solution going, I can 
put the effort where it is most productive, namely into making sure that 
the thread-based solution works best. This is my preferred outcome and 
current goal. :)

Cheers,
Garth