Embedding Python in multi-threaded app

[Harri Pesonen]

Hello. Thanks to the message below
(http://groups.google.fi/groups?selm=dkuhlmanF1KAr0.5J9%40netcom.com),
I was able to do the same, almost. I have currently only one problem.
When I call Py_Finalize() when the dll terminates, it crashes. I have
Python 2.3. Everything else works, I have run many different scripts
in threads. I wonder if someone has noticed anything similar, and
perhaps found a solution.
 
Also I don't understand PyThreadState_Swap(interp) call below, but it
does not work properly without it, the thread termination hangs
without it. The documentation says that Py_NewInterpreter() already
makes the new state current.
 
Thanks for any help!

Harri

--- Original message below ----
 
Thanks for comments from Guido. I have a very preliminary version 
working with Python embedded in a multi-threaded C++ application.

I'd like to summarize what I'm doing so that others will have an
example to follow or so that someone can correct me if I'm still
doing something wrong.

My application creates multiple threads, one for each end-user
session. I create a separate Python sub-interpreter for each thread
(session), then evaluate scripts in that session with the
sub-interpreter for that session.  I do this so that within a
session, scripts are evaluated under a context/environment local to
that session and so that the scripts in a session do not share
(borrow or steal) global values, imported functions, etc from other
sessions. Thus, my users (who are developing Python scripts) can
write scripts from the viewpoint that the scripts for each session
will run in their own little world under their own separate
environment. I view this as a very important benefit that embedded
Python gives me.

Here is a summary of my latest code:

  // At the very beginning of the application.
  Py_Initialize();
  PyEval_InitThreads();
  PyEval_ReleaseLock();
    o
    o
    o
  // Here we are in the constructor for a session.
  PyEval_AcquireLock();
  interp = Py_NewInterpreter();
  // Note that this application has multiple threads.
  // It maintains a separate interp (sub-interpreter) for each thread.
  PyThreadState_Swap(interp);
  // Call some Python C/API functions here
  PyEval_ReleaseThread(interp);
    o
    o
    o
  // Get sub-interpreter for current thread.
  interp = ...
  PyEval_AcquireThread(interp);
  // Call more Python C/API functions here, e.g. PyRun_SimpleString
  PyEval_ReleaseThread(interp);
    o
    o
    o
  // Get sub-interpreter for current thread.
  interp = ...
  PyEval_AcquireThread(interp);
  // Call more Python C/API functions here
  PyEval_ReleaseThread(interp);
    o
    o
    o
  // Here we are in the destructor for a session.
  // Get sub-interpreter for current thread.
  interp = ...
  PyEval_AcquireThread(interp);
  Py_EndInterpreter(interp);
  PyEval_ReleaseLock();
    o
    o
    o
  // At the very end of the application.
  PyEval_AcquireLock();
  Py_Finalize();


Thanks again for help.  And I hope that this can serve as a simple
model for others trying to embed Python in a multi-threaded
application.

  -- Dave



Guido van Rossum <guido at CNRI.Reston.VA.US> wrote:

> Yes, but this is still dangerous.  You should really use the APIs that 
> manage the lock *and* the thread state: use
> PyEval_AcquireThread() and PyEval_ReleaseThread().  (The argument to
> the latter should be the current thread state, gotten from
> PyThreadState_Get().) 
> In case you're inclined to continue this while I'm unavailable, read
> the source -- e.g. the routines I mentioned here are in
> Python/ceval.c. 
> --Guido van Rossum (home page: http://www.python.org/~guido/)