2.6, 3.0, and truly independent intepreters

["Martin v. Löwis"]

>> Why do you think so? For C code that is carefully written, the GIL
>> allows *very well* to write CPU bound scripts running on other threads.
>> (please do get back to Jesse's original remark in case you have lost
>> the thread :-)
>>
> 
> I don't follow you there.  If you're referring to multiprocessing

No, I'm not. I refer to regular, plain, multi-threading.

>>> It's turns out that this isn't an exotic case
>>> at all: there's a *ton* of utility gained by making calls back into
>>> the interpreter. The best example is that since code more easily
>>> maintained in python than in C, a lot of the module "utility" code is
>>> likely to be in python.
>> You should really reconsider writing performance-critical code in
>> Python.
> 
> I don't follow you there...  Performance-critical code in Python??

I probably expressed myself incorrectly (being not a native speaker
of English): If you were writing performance-critical in Python,
you should reconsider (i.e. you should rewrite it in C).

It's not clear whether this calling back into Python is in the
performance-critical path. If it is, then reconsider.

> I tried to list some abbreviated examples in other posts, but here's
> some elaboration:
> 
> - Pixel-level effects and filters, where some filters may use C procs
> while others may call back into the interpreter to execute logic --
> while some do both, multiple times.

Ok. For a plain C proc, release the GIL before the proc, and reacquire
it afterwards. For a proc that calls into the interpreter:
a) if it is performance-critical, reconsider writing it in C, or
   reformulate so that it stops being performance critical (e.g.
   through caching)
b) else, reacquire the GIL before calling back into Python, then
   release the GIL before continuing the proc

> - Image and video analysis/recognition where there's TONS of intricate
> data structures and logic.  Those data structures and logic are
> easiest to develop and maintain in python, but you'll often want to
> call back to C procs which will, in turn, want to access Python (as
> well as C-level) data structures.

Not sure what the processing is, or what processing you need to do.
The data structures themselves are surely not performance critical
(not being algorithms). If you really run Python algorithms on these
structures, then my approach won't help you (except for the general
recommendation to find some expensive sub-algorithm and rewrite that
in C, so that it both becomes faster and can release the GIL).

> It's just not practical to be
> locking and locking the GIL when you want to operate on python data
> structures or call back into python.

This I don't understand. I find that fairly easy to do.

> You seem to have placed the burden of proof on my shoulders for an app
> to deserve the ability to free-thread when using 3rd party packages,
> so how about we just agree it's not an unreasonable desire for a
> package (such as python) to support it and move on with the
> discussion.

Not at all - I don't want a proof. I just want agreement on Jesse
Noller's claim

# A c-level module, on the other hand, can sidestep/release
# the GIL at will, and go on it's merry way and process away.

>> If neither is likely to result, killing the discussion is the most
>> productive thing we can do.
>>
> 
> Well, most others here seem to have a lot different definition of what
> qualifies as a "futile" discussion, so how about you allow the rest of
> us continue to discuss these issues and possible solutions.  And, for
> the record, I've said multiple times I'm ready to contribute
> monetarily, professionally, and personally, so if that doesn't qualify
> as the precursor to "code contributions from one of the participants"
> then I don't know WHAT does.

Ok, I apologize for having misunderstood you here.

Regards,
Martin