[IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)

[Dino Viehland]

Thanks for the info...  Darn, it sounds like you actually want sys to be isolated...

We're going to do a review over the hosting APIs internally next week.   That should result in the last large set of changes.  One thought that has occurred to me is the ability to provide management of the current sys instance.  This is where most of the state is tracked.  That would presumably be something like allowing you to create a new system state and provide the current system state at runtime.  You could then switch system states before running a new script code.  In the multi-threaded case you could store the system state in a thread local variable.  Therefore rather than tracking a set of engines you'd track a list of SystemState objects (or in DLR speak they may be some opaque state object).

The one interesting thing here is how this plays w/ console output.  In v2.0 that's separate from SystemState so you'd also need to update those at the same time.

Any feedback on that proposal?



-----Original Message-----
From: users-bounces at lists.ironpython.com [mailto:users-bounces at lists.ironpython.com] On Behalf Of Tony Djordjevski
Sent: Wednesday, May 02, 2007 9:59 AM
To: Discussion of IronPython
Subject: Re: [IronPython] Multiple engine instances in IP 2.0 and beyond (was IronPython 2.0 Alpha 1 Released)

Thanks for the reply Dino.  And sorry, this is going to be a long post.

I guess the best way to tell you what we need from IronPython is to tell
you how our product works today.  I should also note that we're porting
the application from native C++ to .Net.  We currently have python
support by way of CPython and the Boost Python library.

Our product is an emulation/simulation application within the
manufacturing domain.  Our customers are able to emulate their
manufacturing process by adding virtual machines (what we call
components) to a 3D world that represents their factory floor.  Each
component can have various "behaviors", and we supply some common
behaviors which the customer can add to a component.  For example, a
conveyor component would have behaviors that deal with conveyance,
sensors that detect parts moving along the conveyor, and signals used
for communication, among others.

The scripts are important because they are a means for the customer to
create custom behaviors (you can almost think of them as plugins).
Using the above example of a conveyor, a part might move down the
conveyor, a sensor would detect that part and then fire a signal.  That
signal could be "wired" up to a script behavior.  The script in turn
might turn off the conveyor, or slow it down, or do throughput analysis,
or a number of other things.

The customer might have a number of these same conveyors on the factory
floor all with the same script behavior(s).  The scripts need to be
isolated insofar that when they do turn off the conveyor, as in the
example above, that they turn off the conveyor in the component they are
associated with.  So I would have to say that the state of the script
"instance" needs to be separate from each other.

The scripts don't run all the time.  They run once during compilation,
as that's the behavior of CPython.  So any statements at the main module
level get executed then.  If the script has specific functions defined,
we call those python functions when particular simulation events take place.

To answer some of your specific questions:
Q1. Do you need multiple system states so they get their own modules,
console, etc...  do you need everything in sys isolated?

A1. I would have to say yes, for the most part.  Shallow copies of
imported modules are probably OK, but we aren't far enough into the
migration to give you a definitive answer.  There is one case in our C++
implementation that this caused a problem and we were forced to call the
module's init function for each interpreter behind the scenes.

As far as everything in sys is concerned, maybe not but I can think of
at least one situation that it must be isolated.  Currently, we redirect
sys.stdout and sys.stderr to a window in our application, but that
doesn't stop a user from redefining those manually in one of their
scripts.  If they do, then it shouldn't affect any other script in the
application.  Other things, like sys.version, don't have to be isolated.

Q2. Do you need to guarantee the isolation even if .NET code is called
(e.g. they could smuggled data via a static field).

A2. No and in our case I don't think that would be desirable.  The
scripts themselves need to be able to interact with other objects within
the simulation world.

Q3. If they do need some rather high level of isolation are app domains
good enough?  Do you need to marshal a lot of data in/out?  Or is the
effort to spin up and use app domains correctly?

A4. I'm fairly new to .Net, so I'm not sure.  I'll have to read up on
app domains, but my gut tells me that is not what we want here.


I hope this sheds more light, Dino.  I tried to be as clear as possible,
but it was a bit difficult to keep this short(ish) and still convey our
needs.

Let me know if I can elaborate on any other questions you might have, or
if anything I've wrote here is unclear.

One thing to think about: eventually we want to make our simulation
engine multi-threaded.  Script functions will probably be getting
executed in parallel at some point.

Thanks,
Tony

Dino Viehland wrote:
> I'm not actually the one working on the engine APIs so that's the reason I've tended to be vague.  I'll talk to the people doing it and let you know what I hear.
>
> But the more info you can give us the better decision we'll be able to make.  For example what do you actually need to be isolated?  Do you need multiple system states so they get their own modules, console, etc...  do you need everything in sys isolated?  Do you need to guarantee the isolation even if .NET code is called (e.g. they could smuggled data via a static field).  If they do need some rather high level of isolation are app domains good enough?  Do you need to marshal a lot of data in/out?  Or is the effort to spin up and use app domains correctly?
>
> Ultimately understanding the scenarios where this is being used might end up with us coming up with an even better solution.
>
> The reasons why I ask that is because there are already many options on isolation depending upon what you're trying to do.  For example in the dynamic language support for ASP.NET we're executing the same code in multiple modules.  That gives each thread it's own set of globals but could allow sharing through system state.  It also means that the different requests get to share imported modules.  That's not much isolation but it could be enough for a lot of scenarios.  On the other extreme is full app domain isolation.
>
> The more info we get here the more likely we'll capture the right result.  The real challenge here is it's actually impossible for us to offer full isolation w/o going to app domain isolation.  So anything we create will be less;  for example Michael Foord noted that the recursion enforcement is a global setting and it's hard to not make that global.  So we just need to figure out the right "less" that is both useful, maintainable, and understandable.
>
_______________________________________________
users mailing list
users at lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com