2.6, 3.0, and truly independent intepreters

[Andy O'Meara]

On Oct 24, 2:12 am, greg <g... at cosc.canterbury.ac.nz> wrote:
> Andy wrote:
> > 1) Independent interpreters (this is the easier one--and solved, in
> > principle anyway, by PEP 3121, by Martin v. Löwis
>
> Something like that is necessary for independent interpreters,
> but not sufficient. There are also all the built-in constants
> and type objects to consider. Most of these are statically
> allocated at the moment.
>

Agreed--I  was just trying to speak generally.  Or, put another way,
there's no hope for independent interpreters without the likes of PEP
3121.  Also, as Martin pointed out, there's the issue of module
cleanup some guys here may underestimate (and I'm glad Martin pointed
out the importance of it).  Without the module cleanup, every time a
dynamic library using python loads and unloads you've got leaks.  This
issue is a real problem for us since our software is loaded and
unloaded many many times in a host app (iTunes, WMP, etc).  I hadn't
raised it here yet (and I don't want to turn the discussion to this),
but lack of multiple load and unload support has been another painful
issue that we didn't expect to encounter when we went with python.


> > 2) Barriers to "free threading".  As Jesse describes, this is simply
> > just the GIL being in place, but of course it's there for a reason.
> > It's there because (1) doesn't hold and there was never any specs/
> > guidance put forward about what should and shouldn't be done in multi-
> > threaded apps
>
> No, it's there because it's necessary for acceptable performance
> when multiple threads are running in one interpreter. Independent
> interpreters wouldn't mean the absence of a GIL; it would only
> mean each interpreter having its own GIL.
>

I see what you're saying, but let's note that what you're talking
about at this point is an interpreter containing protection from the
client level violating (supposed) direction put forth in python
multithreaded guidelines.  Glenn Linderman's post really gets at
what's at hand here.  It's really important to consider that it's not
a given that python (or any framework) has to be designed against
hazardous use.  Again, I refer you to the diagrams and guidelines in
the QuickTime API:

http://developer.apple.com/technotes/tn/tn2125.html

They tell you point-blank what you can and can't do, and it's that's
simple.  Their engineers can then simply create the implementation
around those specs and not weigh any of the implementation down with
sync mechanisms.  I'm in the camp that simplicity and convention wins
the day when it comes to an API.  It's safe to say that software
engineers expect and assume that a thread that doesn't have contact
with other threads (except for explicit, controlled message/object
passing) will run unhindered and safely, so I raise an eyebrow at the
GIL (or any internal "helper" sync stuff) holding up an thread's
performance when the app is designed to not need lower-level global
locks.

Anyway, let's talk about solutions.  My company looking to support
python dev community endeavor that allows the following:

- an app makes N worker threads (using the OS)

- each worker thread makes its own interpreter, pops scripts off a
work queue, and manages exporting (and then importing) result data to
other parts of the app.  Generally, we're talking about CPU-bound work
here.

- each interpreter has the essentials (e.g. math support, string
support, re support, and so on -- I realize this is open-ended, but
work with me here).

Let's guesstimate about what kind of work we're talking about here and
if this is even in the realm of possibility.  If we find that it *is*
possible, let's figure out what level of work we're talking about.
>From there, I can get serious about writing up a PEP/spec, paid
support, and so on.

Regards,
Andy