HELP! Obscure Python Reentrancy Question

[David Bolen]

Courageous <jkraska at san.rr.com> writes:

> 3. The C++ program occasionally dispatches to a bit of Python
> code.
> 
> 4. The Python code calls an external C++ function which
> triggers the thread switch back to the scheduler.
> 
> 5. The scheduler starts or resumes another thread, which
> very likely will contain more Python, some of which has
> been left in this intermediate state like this.
> 
> My test shows this going to hell in a handbasket. That
> doesn't surprise me.
> 
> What now?

I know this was designed as a minimal description, but a little more
detail might be helpful.

How have you allocated interpreter state(s) versus your C++ scheduler
threads (or fibers) and how are you maintaining the GIL?

When the other thread in 5 resumes and "contains" more Python, does
that mean that from the perspective of the Python script you are
"returning" from the call in 4, or does that mean there's some
independent Python code you are just starting to execute?

It seems to me that depending on how you want to link your Python code
execution with your scheduler thread states, you may want to have an
interpreter (or at least thread state) per thread, even if your
threads aren't technically native but fibers within a single native
thread.  

In that case, when the currently executing thread that is in Python
code calls into the C++ function and provides the opportunity to yield
to another thread, that new thread can simply continue executing with
its own state - presumably returning to Python from an earlier
function call which itself yielded.

But it's hard to suppose too much more in the abstract.

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/