[IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)

[Tony Djordjevski]

Thanks for the reply Dino.  And sorry, this is going to be a long post.

I guess the best way to tell you what we need from IronPython is to tell 
you how our product works today.  I should also note that we're porting 
the application from native C++ to .Net.  We currently have python 
support by way of CPython and the Boost Python library.

Our product is an emulation/simulation application within the 
manufacturing domain.  Our customers are able to emulate their 
manufacturing process by adding virtual machines (what we call 
components) to a 3D world that represents their factory floor.  Each 
component can have various "behaviors", and we supply some common 
behaviors which the customer can add to a component.  For example, a 
conveyor component would have behaviors that deal with conveyance, 
sensors that detect parts moving along the conveyor, and signals used 
for communication, among others.

The scripts are important because they are a means for the customer to 
create custom behaviors (you can almost think of them as plugins). 
Using the above example of a conveyor, a part might move down the 
conveyor, a sensor would detect that part and then fire a signal.  That 
signal could be "wired" up to a script behavior.  The script in turn 
might turn off the conveyor, or slow it down, or do throughput analysis, 
or a number of other things.

The customer might have a number of these same conveyors on the factory 
floor all with the same script behavior(s).  The scripts need to be 
isolated insofar that when they do turn off the conveyor, as in the 
example above, that they turn off the conveyor in the component they are 
associated with.  So I would have to say that the state of the script 
"instance" needs to be separate from each other.

The scripts don't run all the time.  They run once during compilation, 
as that's the behavior of CPython.  So any statements at the main module 
level get executed then.  If the script has specific functions defined, 
we call those python functions when particular simulation events take place.

To answer some of your specific questions:
Q1. Do you need multiple system states so they get their own modules, 
console, etc...  do you need everything in sys isolated?

A1. I would have to say yes, for the most part.  Shallow copies of 
imported modules are probably OK, but we aren't far enough into the 
migration to give you a definitive answer.  There is one case in our C++ 
implementation that this caused a problem and we were forced to call the 
module's init function for each interpreter behind the scenes.

As far as everything in sys is concerned, maybe not but I can think of 
at least one situation that it must be isolated.  Currently, we redirect 
sys.stdout and sys.stderr to a window in our application, but that 
doesn't stop a user from redefining those manually in one of their 
scripts.  If they do, then it shouldn't affect any other script in the 
application.  Other things, like sys.version, don't have to be isolated.

Q2. Do you need to guarantee the isolation even if .NET code is called 
(e.g. they could smuggled data via a static field).

A2. No and in our case I don't think that would be desirable.  The 
scripts themselves need to be able to interact with other objects within 
the simulation world.

Q3. If they do need some rather high level of isolation are app domains 
good enough?  Do you need to marshal a lot of data in/out?  Or is the 
effort to spin up and use app domains correctly?

A4. I'm fairly new to .Net, so I'm not sure.  I'll have to read up on 
app domains, but my gut tells me that is not what we want here.


I hope this sheds more light, Dino.  I tried to be as clear as possible, 
but it was a bit difficult to keep this short(ish) and still convey our 
needs.

Let me know if I can elaborate on any other questions you might have, or 
if anything I've wrote here is unclear.

One thing to think about: eventually we want to make our simulation 
engine multi-threaded.  Script functions will probably be getting 
executed in parallel at some point.

Thanks,
Tony

Dino Viehland wrote:
> I'm not actually the one working on the engine APIs so that's the reason I've tended to be vague.  I'll talk to the people doing it and let you know what I hear.
> 
> But the more info you can give us the better decision we'll be able to make.  For example what do you actually need to be isolated?  Do you need multiple system states so they get their own modules, console, etc...  do you need everything in sys isolated?  Do you need to guarantee the isolation even if .NET code is called (e.g. they could smuggled data via a static field).  If they do need some rather high level of isolation are app domains good enough?  Do you need to marshal a lot of data in/out?  Or is the effort to spin up and use app domains correctly?
> 
> Ultimately understanding the scenarios where this is being used might end up with us coming up with an even better solution.
> 
> The reasons why I ask that is because there are already many options on isolation depending upon what you're trying to do.  For example in the dynamic language support for ASP.NET we're executing the same code in multiple modules.  That gives each thread it's own set of globals but could allow sharing through system state.  It also means that the different requests get to share imported modules.  That's not much isolation but it could be enough for a lot of scenarios.  On the other extreme is full app domain isolation.
> 
> The more info we get here the more likely we'll capture the right result.  The real challenge here is it's actually impossible for us to offer full isolation w/o going to app domain isolation.  So anything we create will be less;  for example Michael Foord noted that the recursion enforcement is a global setting and it's hard to not make that global.  So we just need to figure out the right "less" that is both useful, maintainable, and understandable.
>