Embedding multiple interpreters

[Garthy]

Hi Chris,

On 06/12/13 19:03, Chris Angelico wrote:
 > On Fri, Dec 6, 2013 at 6:59 PM, Garthy
 > <garthy_nhtyp at entropicsoftware.com>  wrote:
 >> Hi Chris (and Michael),
 >
 > Hehe. People often say that to me IRL, addressing me and my brother.
 > But he isn't on python-list, so you clearly mean Michael Torrie, yet
 > my brain still automatically thought you were addressing Michael
 > Angelico :)

These strange coincidences happen from time to time- it's entertaining 
when they do. :)

 >> To allow each script to run in its own environment, with minimal 
chance of
 >> inadvertent interaction between the environments, whilst allowing each
 >> script the ability to stall on conditions that will be later met by 
another
 >> thread supplying the information, and to fit in with existing
 >> infrastructure.
 >
 > Are the scripts written cooperatively, or must you isolate one from
 > another? If you need to isolate them for trust reasons, then there's
 > only one solution, and that's separate processes with completely
 > separate interpreters. But if you're prepared to accept that one
 > thread of execution is capable of mangling another's state, things are
 > a lot easier. You can protect against *inadvertent* interaction much
 > more easily than malicious interference. It may be that you can get
 > away with simply running multiple threads in one interpreter;
 > obviously that would have problems if you need more than one CPU core
 > between them all (hello GIL), but that would really be your first
 > limit. One thread could fiddle with __builtins__ or a standard module
 > and thus harass another thread, but you would know if that's what's
 > going on.

I think the ideal is completely sandboxed, but it's something that I 
understand I may need to make compromises on. The bare minimum would be 
protection against inadvertent interaction. Better yet would be a setup 
that made such interaction annoyingly difficult, and the ideal would be 
where it was impossible to interfere. My approaching this problem with 
interpreters was based on an assumption that it might provide a 
reasonable level of isolation- perhaps not ideal, but hopefully good enough.

The closest analogy for understanding would be browser plugins: Scripts 
from multiple authors who for the most part aren't looking to create 
deliberate incompatibilities or interference between plugins. The 
isolation is basic, and some effort is made to make sure that one plugin 
can't cripple another trivially, but the protection is not exhaustive.

Strangely enough, the GIL restriction isn't a big one in this case. For 
the application, the common case is actually one script running at a 
time, with other scripts waiting or not running at that time. They do 
sometimes overlap, but this isn't the common case. If it turned out that 
only one script could be progressing at a time, it's an annoyance but 
not a deal-breaker. If it's suboptimal (as seems to be the case), then 
it's actually not a major issue.

With the single interpreter and multiple thread approach suggested, do 
you know if this will work with threads created externally to Python, 
ie. if I can create a thread in my application as normal, and then call 
something like PyGILState_Ensure() to make sure that Python has the 
internals it needs to work with it, and then use the GIL (or similar) to 
ensure that accesses to it remain thread-safe? If the answer is yes I 
can integrate such a thing more easily as an experiment. If it requires 
calling a dedicated "control" script that feeds out threads then it 
would need a fair bit more mucking about to integrate- I'd like to avoid 
this if possible.

Cheers,
Garth