Embedding Python in multi-threaded app

[Harri Pesonen]

I solved it. Downloaded the PyWM source code (http://pywx.idyll.org/)
and checked how it does it. Even if it is linux code, my Win32
application started working (or in fact stopping) correctly.

The correct startup sequence is:

	Py_Initialize();
	PyEval_InitThreads();
	mainState = PyThreadState_Swap(NULL);
	PyEval_ReleaseLock();

And the correct cleanup sequence is:

	PyEval_AcquireLock();
	PyThreadState_Swap(mainState);
	PyEval_ReleaseLock();
	Py_Finalize();

I don't understand why

	mainState = PyThreadState_Swap(NULL);
and
	PyThreadState_Swap(mainState);
	PyEval_ReleaseLock();

are needed, so if someone can explain? :-)

Harri

Harri.Pesonen at wicom.com (Harri Pesonen) wrote in message news:<17fbc98a.0309040028.6776d426 at posting.google.com>...
> Hello. Thanks to the message below
> (http://groups.google.fi/groups?selm=dkuhlmanF1KAr0.5J9%40netcom.com),
> I was able to do the same, almost. I have currently only one problem.
> When I call Py_Finalize() when the dll terminates, it crashes. I have
> Python 2.3. Everything else works, I have run many different scripts
> in threads. I wonder if someone has noticed anything similar, and
> perhaps found a solution.
>  
> Also I don't understand PyThreadState_Swap(interp) call below, but it
> does not work properly without it, the thread termination hangs
> without it. The documentation says that Py_NewInterpreter() already
> makes the new state current.
>  
> Thanks for any help!
> 
> Harri
> 
> --- Original message below ----
>  
> Thanks for comments from Guido. I have a very preliminary version 
> working with Python embedded in a multi-threaded C++ application.
> 
> I'd like to summarize what I'm doing so that others will have an
> example to follow or so that someone can correct me if I'm still
> doing something wrong.
> 
> My application creates multiple threads, one for each end-user
> session. I create a separate Python sub-interpreter for each thread
> (session), then evaluate scripts in that session with the
> sub-interpreter for that session.  I do this so that within a
> session, scripts are evaluated under a context/environment local to
> that session and so that the scripts in a session do not share
> (borrow or steal) global values, imported functions, etc from other
> sessions. Thus, my users (who are developing Python scripts) can
> write scripts from the viewpoint that the scripts for each session
> will run in their own little world under their own separate
> environment. I view this as a very important benefit that embedded
> Python gives me.
> 
> Here is a summary of my latest code:
> 
>   // At the very beginning of the application.
>   Py_Initialize();
>   PyEval_InitThreads();
>   PyEval_ReleaseLock();
>     o
>     o
>     o
>   // Here we are in the constructor for a session.
>   PyEval_AcquireLock();
>   interp = Py_NewInterpreter();
>   // Note that this application has multiple threads.
>   // It maintains a separate interp (sub-interpreter) for each thread.
>   PyThreadState_Swap(interp);
>   // Call some Python C/API functions here
>   PyEval_ReleaseThread(interp);
>     o
>     o
>     o
>   // Get sub-interpreter for current thread.
>   interp = ...
>   PyEval_AcquireThread(interp);
>   // Call more Python C/API functions here, e.g. PyRun_SimpleString
>   PyEval_ReleaseThread(interp);
>     o
>     o
>     o
>   // Get sub-interpreter for current thread.
>   interp = ...
>   PyEval_AcquireThread(interp);
>   // Call more Python C/API functions here
>   PyEval_ReleaseThread(interp);
>     o
>     o
>     o
>   // Here we are in the destructor for a session.
>   // Get sub-interpreter for current thread.
>   interp = ...
>   PyEval_AcquireThread(interp);
>   Py_EndInterpreter(interp);
>   PyEval_ReleaseLock();
>     o
>     o
>     o
>   // At the very end of the application.
>   PyEval_AcquireLock();
>   Py_Finalize();
> 
> 
> Thanks again for help.  And I hope that this can serve as a simple
> model for others trying to embed Python in a multi-threaded
> application.
> 
>   -- Dave
> 
> 
> 
> Guido van Rossum <guido at CNRI.Reston.VA.US> wrote:
> 
> > Yes, but this is still dangerous.  You should really use the APIs that 
> > manage the lock *and* the thread state: use
> > PyEval_AcquireThread() and PyEval_ReleaseThread().  (The argument to
> > the latter should be the current thread state, gotten from
> > PyThreadState_Get().) 
> > In case you're inclined to continue this while I'm unavailable, read
> > the source -- e.g. the routines I mentioned here are in
> > Python/ceval.c. 
> > --Guido van Rossum (home page: http://www.python.org/~guido/)